2024-05-04T12:00:00.192 INFO:root:teuthology version: 0.0.1.dev271+g1ae7ad8 2024-05-04T12:00:00.192 DEBUG:teuthology.run:Teuthology command: teuthology --name yuriw-2024-05-02_19:10:09-fs-wip-yuri4-testing-2024-04-29-0642-distro-default-smithi --description fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.1} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} --verbose --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2024-05-02_19:10:09-fs-wip-yuri4-testing-2024-04-29-0642-distro-default-smithi/7686275 -- /home/teuthworker/archive/yuriw-2024-05-02_19:10:09-fs-wip-yuri4-testing-2024-04-29-0642-distro-default-smithi/7686275/orig.config.yaml 2024-05-04T12:00:00.244 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-05-04T12:00:00.339 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-05-02_19:10:09-fs-wip-yuri4-testing-2024-04-29-0642-distro-default-smithi/7686275 branch: wip-yuri4-testing-2024-04-29-0642 description: fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.1} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} email: null first_in_suite: false job_id: '7686275' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'setup ceph/v18.2.1 ' name: yuriw-2024-05-02_19:10:09-fs-wip-yuri4-testing-2024-04-29-0642-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-yuri4-testing-2024-04-29-0642 ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: mon pg warn min per osd: 0 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - OSD_DOWN - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - MON_DOWN sha1: 09dbd6bba52d19ba471c33bc15c008e6ad158ea6 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 09dbd6bba52d19ba471c33bc15c008e6ad158ea6 kclient: syntax: v1 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri4-testing-2024-04-29-0642 sha1: 09dbd6bba52d19ba471c33bc15c008e6ad158ea6 owner: scheduled_yuriw@teuthology priority: 99 repo: https://github.com/ceph/ceph-ci.git roles: - - host.a - client.0 - osd.0 - osd.1 - osd.2 - - host.b - client.1 - osd.3 - osd.4 - osd.5 seed: 1311 sha1: 09dbd6bba52d19ba471c33bc15c008e6ad158ea6 sleep_before_teardown: 0 subset: 111/120000 suite: fs suite_branch: wip-yuri4-testing-2024-04-29-0642 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_09dbd6bba52d19ba471c33bc15c008e6ad158ea6/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 09dbd6bba52d19ba471c33bc15c008e6ad158ea6 targets: smithi079.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIHcuUETowTw9yhcydIBCjPslg9zshm19XIjmkFU3JrQdIY8ardoSS6N14RADKyAO/YwvweqfkZ6PWZ1qumUh1g= smithi146.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIAm749bFz7aEejwdZa56XYzAwBSFHJAm+KZD1iGYlzY815cjRcRs8hktFJusOtxnspIppe+lxO7fvJzjCBOFzs= tasks: - install: exclude_packages: - ceph-volume tag: v18.2.1 - print: '**** done install task...' - cephadm: compiled_cephadm_branch: reef conf: osd: osd_class_default_list: '*' osd_class_load_list: '*' image: quay.io/ceph/ceph:v18.2.1 roleless: true - print: '**** done end installing v18.2.1 cephadm ...' - cephadm.shell: host.a: - ceph config set mgr mgr/cephadm/use_repo_digest true --force - print: '**** done cephadm.shell ceph config set mgr...' - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: host.a: - ceph fs volume create cephfs --placement=4 - ceph fs dump - cephadm.shell: host.a: - ceph fs set cephfs max_mds 1 - cephadm.shell: host.a: - ceph fs set cephfs allow_standby_replay false - cephadm.shell: host.a: - ceph fs set cephfs inline_data true --yes-i-really-really-mean-it - cephadm.shell: host.a: - ceph fs dump - ceph --format=json fs dump | jq -e ".filesystems | length == 1" - while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done - fs.pre_upgrade_save: null - kclient: null - print: '**** done client' - parallel: - upgrade-tasks - workload-tasks - cephadm.shell: host.a: - ceph fs dump - fs.post_upgrade_checks: null teuthology: fragments_dropped: - /cephfs/home/yuriw/src/github.com_ceph_ceph-c_wip-yuri4-testing-2024-04-29-0642/qa/suites/fs/upgrade/mds_upgrade_sequence/tasks/3-upgrade-mgr-staggered.yaml meta: {} postmerge: [] variables: fail_fs: false teuthology_branch: main teuthology_sha1: 1ae7ad82388e92a475afff437d49054826c019a1 timestamp: 2024-05-02_19:10:09 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mgr mgr/orchestrator/fail_fs false || true - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done - ceph orch ps - ceph orch upgrade status - ceph health detail - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 workload-tasks: sequential: - workunit: clients: all: - suites/fsstress.sh 2024-05-04T12:00:00.339 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_09dbd6bba52d19ba471c33bc15c008e6ad158ea6/qa; will attempt to use it 2024-05-04T12:00:00.340 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_09dbd6bba52d19ba471c33bc15c008e6ad158ea6/qa/tasks 2024-05-04T12:00:00.340 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-05-04T12:00:00.342 INFO:teuthology.task.internal:Checking packages... 2024-05-04T12:00:00.366 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '09dbd6bba52d19ba471c33bc15c008e6ad158ea6' 2024-05-04T12:00:00.366 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-05-04T12:00:00.366 INFO:teuthology.packaging:ref: None 2024-05-04T12:00:00.366 INFO:teuthology.packaging:tag: None 2024-05-04T12:00:00.366 INFO:teuthology.packaging:branch: wip-yuri4-testing-2024-04-29-0642 2024-05-04T12:00:00.366 INFO:teuthology.packaging:sha1: 09dbd6bba52d19ba471c33bc15c008e6ad158ea6 2024-05-04T12:00:00.366 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-yuri4-testing-2024-04-29-0642 2024-05-04T12:00:00.521 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2455.g09dbd6bb 2024-05-04T12:00:00.523 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-05-04T12:00:00.530 INFO:teuthology.task.internal:no buildpackages task found 2024-05-04T12:00:00.530 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-05-04T12:00:00.543 INFO:teuthology.task.internal:Saving configuration 2024-05-04T12:00:00.560 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-05-04T12:00:00.566 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-05-04T12:00:00.599 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi079.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-05-02_19:10:09-fs-wip-yuri4-testing-2024-04-29-0642-distro-default-smithi/7686275', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-04 11:54:57.644276', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIHcuUETowTw9yhcydIBCjPslg9zshm19XIjmkFU3JrQdIY8ardoSS6N14RADKyAO/YwvweqfkZ6PWZ1qumUh1g='} 2024-05-04T12:00:00.630 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi146.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-05-02_19:10:09-fs-wip-yuri4-testing-2024-04-29-0642-distro-default-smithi/7686275', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-04 11:54:57.645732', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIAm749bFz7aEejwdZa56XYzAwBSFHJAm+KZD1iGYlzY815cjRcRs8hktFJusOtxnspIppe+lxO7fvJzjCBOFzs='} 2024-05-04T12:00:00.631 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-05-04T12:00:00.636 INFO:teuthology.task.internal:roles: ubuntu@smithi079.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2024-05-04T12:00:00.636 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2024-05-04T12:00:00.637 INFO:teuthology.run_tasks:Running task console_log... 2024-05-04T12:00:00.744 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7eff79aee160>, signals=[15]) 2024-05-04T12:00:00.744 INFO:teuthology.run_tasks:Running task internal.connect... 2024-05-04T12:00:00.751 INFO:teuthology.task.internal:Opening connections... 2024-05-04T12:00:00.751 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi079.front.sepia.ceph.com 2024-05-04T12:00:00.754 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi079.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-04T12:00:00.854 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi146.front.sepia.ceph.com 2024-05-04T12:00:00.855 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-04T12:00:00.954 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-05-04T12:00:00.961 DEBUG:teuthology.orchestra.run.smithi079:> uname -m 2024-05-04T12:00:00.978 INFO:teuthology.orchestra.run.smithi079.stdout:x86_64 2024-05-04T12:00:00.979 DEBUG:teuthology.orchestra.run.smithi079:> cat /etc/os-release 2024-05-04T12:00:01.036 INFO:teuthology.orchestra.run.smithi079.stdout:NAME="CentOS Stream" 2024-05-04T12:00:01.036 INFO:teuthology.orchestra.run.smithi079.stdout:VERSION="9" 2024-05-04T12:00:01.036 INFO:teuthology.orchestra.run.smithi079.stdout:ID="centos" 2024-05-04T12:00:01.036 INFO:teuthology.orchestra.run.smithi079.stdout:ID_LIKE="rhel fedora" 2024-05-04T12:00:01.036 INFO:teuthology.orchestra.run.smithi079.stdout:VERSION_ID="9" 2024-05-04T12:00:01.036 INFO:teuthology.orchestra.run.smithi079.stdout:PLATFORM_ID="platform:el9" 2024-05-04T12:00:01.036 INFO:teuthology.orchestra.run.smithi079.stdout:PRETTY_NAME="CentOS Stream 9" 2024-05-04T12:00:01.037 INFO:teuthology.orchestra.run.smithi079.stdout:ANSI_COLOR="0;31" 2024-05-04T12:00:01.037 INFO:teuthology.orchestra.run.smithi079.stdout:LOGO="fedora-logo-icon" 2024-05-04T12:00:01.037 INFO:teuthology.orchestra.run.smithi079.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-05-04T12:00:01.037 INFO:teuthology.orchestra.run.smithi079.stdout:HOME_URL="https://centos.org/" 2024-05-04T12:00:01.037 INFO:teuthology.orchestra.run.smithi079.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-05-04T12:00:01.037 INFO:teuthology.orchestra.run.smithi079.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-05-04T12:00:01.037 INFO:teuthology.orchestra.run.smithi079.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-05-04T12:00:01.037 INFO:teuthology.lock.ops:Updating smithi079.front.sepia.ceph.com on lock server 2024-05-04T12:00:01.076 DEBUG:teuthology.orchestra.run.smithi146:> uname -m 2024-05-04T12:00:01.093 INFO:teuthology.orchestra.run.smithi146.stdout:x86_64 2024-05-04T12:00:01.094 DEBUG:teuthology.orchestra.run.smithi146:> cat /etc/os-release 2024-05-04T12:00:01.150 INFO:teuthology.orchestra.run.smithi146.stdout:NAME="CentOS Stream" 2024-05-04T12:00:01.150 INFO:teuthology.orchestra.run.smithi146.stdout:VERSION="9" 2024-05-04T12:00:01.151 INFO:teuthology.orchestra.run.smithi146.stdout:ID="centos" 2024-05-04T12:00:01.151 INFO:teuthology.orchestra.run.smithi146.stdout:ID_LIKE="rhel fedora" 2024-05-04T12:00:01.151 INFO:teuthology.orchestra.run.smithi146.stdout:VERSION_ID="9" 2024-05-04T12:00:01.151 INFO:teuthology.orchestra.run.smithi146.stdout:PLATFORM_ID="platform:el9" 2024-05-04T12:00:01.151 INFO:teuthology.orchestra.run.smithi146.stdout:PRETTY_NAME="CentOS Stream 9" 2024-05-04T12:00:01.151 INFO:teuthology.orchestra.run.smithi146.stdout:ANSI_COLOR="0;31" 2024-05-04T12:00:01.151 INFO:teuthology.orchestra.run.smithi146.stdout:LOGO="fedora-logo-icon" 2024-05-04T12:00:01.151 INFO:teuthology.orchestra.run.smithi146.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-05-04T12:00:01.151 INFO:teuthology.orchestra.run.smithi146.stdout:HOME_URL="https://centos.org/" 2024-05-04T12:00:01.151 INFO:teuthology.orchestra.run.smithi146.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-05-04T12:00:01.151 INFO:teuthology.orchestra.run.smithi146.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-05-04T12:00:01.151 INFO:teuthology.orchestra.run.smithi146.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-05-04T12:00:01.151 INFO:teuthology.lock.ops:Updating smithi146.front.sepia.ceph.com on lock server 2024-05-04T12:00:01.184 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-05-04T12:00:01.193 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-05-04T12:00:01.199 INFO:teuthology.task.internal:Checking for old test directory... 2024-05-04T12:00:01.199 DEBUG:teuthology.orchestra.run.smithi079:> test '!' -e /home/ubuntu/cephtest 2024-05-04T12:00:01.201 DEBUG:teuthology.orchestra.run.smithi146:> test '!' -e /home/ubuntu/cephtest 2024-05-04T12:00:01.217 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-05-04T12:00:01.223 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-05-04T12:00:01.223 DEBUG:teuthology.orchestra.run.smithi079:> test -z $(ls -A /var/lib/ceph) 2024-05-04T12:00:01.257 DEBUG:teuthology.orchestra.run.smithi146:> test -z $(ls -A /var/lib/ceph) 2024-05-04T12:00:01.281 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-05-04T12:00:01.354 INFO:teuthology.run_tasks:Running task kernel... 2024-05-04T12:00:01.368 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-05-04T12:00:01.368 INFO:teuthology.task.kernel:config {'host.a': {'kdb': 1, 'sha1': 'distro'}, 'host.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-05-04T12:00:01.368 DEBUG:teuthology.orchestra.run.smithi079:> test -f /run/.containerenv -o -f /.dockerenv 2024-05-04T12:00:01.369 DEBUG:teuthology.orchestra.run.smithi146:> test -f /run/.containerenv -o -f /.dockerenv 2024-05-04T12:00:01.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-04T12:00:01.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-04T12:00:01.385 DEBUG:teuthology.orchestra.run.smithi079:> uname -r 2024-05-04T12:00:01.385 DEBUG:teuthology.orchestra.run.smithi146:> uname -r 2024-05-04T12:00:01.441 INFO:teuthology.orchestra.run.smithi079.stdout:5.14.0-437.el9.x86_64 2024-05-04T12:00:01.441 INFO:teuthology.task.kernel:Running kernel on smithi079: 5.14.0-437.el9.x86_64 2024-05-04T12:00:01.441 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum install -y kernel 2024-05-04T12:00:01.442 INFO:teuthology.orchestra.run.smithi146.stdout:5.14.0-437.el9.x86_64 2024-05-04T12:00:01.442 INFO:teuthology.task.kernel:Running kernel on smithi146: 5.14.0-437.el9.x86_64 2024-05-04T12:00:01.442 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum install -y kernel 2024-05-04T12:00:03.818 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.0 MB 00:00 2024-05-04T12:00:05.338 INFO:teuthology.orchestra.run.smithi079.stdout:CentOS Stream 9 - BaseOS 3.7 MB/s | 8.0 MB 00:02 2024-05-04T12:00:07.027 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS Stream 9 - AppStream 8.8 MB/s | 19 MB 00:02 2024-05-04T12:00:09.013 INFO:teuthology.orchestra.run.smithi079.stdout:CentOS Stream 9 - AppStream 7.3 MB/s | 19 MB 00:02 2024-05-04T12:00:11.328 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS Stream 9 - CRB 9.3 MB/s | 6.2 MB 00:00 2024-05-04T12:00:12.972 INFO:teuthology.orchestra.run.smithi079.stdout:CentOS Stream 9 - CRB 15 MB/s | 6.2 MB 00:00 2024-05-04T12:00:13.037 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS Stream 9 - Extras packages 44 kB/s | 16 kB 00:00 2024-05-04T12:00:14.514 INFO:teuthology.orchestra.run.smithi079.stdout:CentOS Stream 9 - Extras packages 84 kB/s | 16 kB 00:00 2024-05-04T12:00:14.524 INFO:teuthology.orchestra.run.smithi146.stdout:Extra Packages for Enterprise Linux 15 MB/s | 21 MB 00:01 2024-05-04T12:00:16.417 INFO:teuthology.orchestra.run.smithi079.stdout:Extra Packages for Enterprise Linux 12 MB/s | 21 MB 00:01 2024-05-04T12:00:19.662 INFO:teuthology.orchestra.run.smithi146.stdout:lab-extras 11 kB/s | 1.7 kB 00:00 2024-05-04T12:00:21.512 INFO:teuthology.orchestra.run.smithi146.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-05-04T12:00:21.513 INFO:teuthology.orchestra.run.smithi146.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-05-04T12:00:21.540 INFO:teuthology.orchestra.run.smithi079.stdout:lab-extras 14 kB/s | 1.7 kB 00:00 2024-05-04T12:00:21.607 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-05-04T12:00:21.608 INFO:teuthology.orchestra.run.smithi146.stdout:Nothing to do. 2024-05-04T12:00:21.608 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-05-04T12:00:21.761 DEBUG:teuthology.orchestra.run.smithi146:> echo no | sudo yum reinstall kernel || true 2024-05-04T12:00:22.264 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:00:03 ago on Sat 04 May 2024 12:00:19 PM UTC. 2024-05-04T12:00:22.395 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-05-04T12:00:22.404 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-05-04T12:00:22.404 INFO:teuthology.orchestra.run.smithi146.stdout: Package Architecture Version Repository Size 2024-05-04T12:00:22.404 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-05-04T12:00:22.404 INFO:teuthology.orchestra.run.smithi146.stdout:Reinstalling: 2024-05-04T12:00:22.404 INFO:teuthology.orchestra.run.smithi146.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-05-04T12:00:22.404 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-05-04T12:00:22.404 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-05-04T12:00:22.404 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-05-04T12:00:22.404 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-05-04T12:00:22.404 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 255 k 2024-05-04T12:00:22.405 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 0 2024-05-04T12:00:22.405 INFO:teuthology.orchestra.run.smithi146.stderr:Operation aborted. 2024-05-04T12:00:22.459 INFO:teuthology.orchestra.run.smithi146.stdout:Is this ok [y/N]: 2024-05-04T12:00:22.459 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum reinstall -y kernel || true 2024-05-04T12:00:22.959 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:00:03 ago on Sat 04 May 2024 12:00:19 PM UTC. 2024-05-04T12:00:23.075 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-05-04T12:00:23.076 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-05-04T12:00:23.076 INFO:teuthology.orchestra.run.smithi146.stdout: Package Architecture Version Repository Size 2024-05-04T12:00:23.076 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-05-04T12:00:23.076 INFO:teuthology.orchestra.run.smithi146.stdout:Reinstalling: 2024-05-04T12:00:23.076 INFO:teuthology.orchestra.run.smithi146.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-05-04T12:00:23.076 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-05-04T12:00:23.076 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-05-04T12:00:23.077 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-05-04T12:00:23.077 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-05-04T12:00:23.077 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 255 k 2024-05-04T12:00:23.077 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 0 2024-05-04T12:00:23.077 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-05-04T12:00:23.388 INFO:teuthology.orchestra.run.smithi079.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-05-04T12:00:23.388 INFO:teuthology.orchestra.run.smithi079.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-05-04T12:00:23.479 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-05-04T12:00:23.480 INFO:teuthology.orchestra.run.smithi079.stdout:Nothing to do. 2024-05-04T12:00:23.480 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-05-04T12:00:23.596 DEBUG:teuthology.orchestra.run.smithi079:> echo no | sudo yum reinstall kernel || true 2024-05-04T12:00:23.908 INFO:teuthology.orchestra.run.smithi146.stdout:kernel-5.14.0-437.el9.x86_64.rpm 362 kB/s | 255 kB 00:00 2024-05-04T12:00:23.909 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-05-04T12:00:23.909 INFO:teuthology.orchestra.run.smithi146.stdout:Total 306 kB/s | 255 kB 00:00 2024-05-04T12:00:23.930 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-05-04T12:00:24.099 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:00:03 ago on Sat 04 May 2024 12:00:21 PM UTC. 2024-05-04T12:00:24.110 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-05-04T12:00:24.111 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-05-04T12:00:24.116 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-05-04T12:00:24.116 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-05-04T12:00:24.219 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-05-04T12:00:24.251 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-05-04T12:00:24.275 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-05-04T12:00:24.275 INFO:teuthology.orchestra.run.smithi079.stdout: Package Architecture Version Repository Size 2024-05-04T12:00:24.275 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-05-04T12:00:24.275 INFO:teuthology.orchestra.run.smithi079.stdout:Reinstalling: 2024-05-04T12:00:24.275 INFO:teuthology.orchestra.run.smithi079.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-05-04T12:00:24.275 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-05-04T12:00:24.275 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-05-04T12:00:24.275 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-05-04T12:00:24.276 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-05-04T12:00:24.276 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 255 k 2024-05-04T12:00:24.276 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 0 2024-05-04T12:00:24.276 INFO:teuthology.orchestra.run.smithi079.stderr:Operation aborted. 2024-05-04T12:00:24.325 INFO:teuthology.orchestra.run.smithi146.stdout: Reinstalling : kernel-5.14.0-437.el9.x86_64 1/2 2024-05-04T12:00:24.326 INFO:teuthology.orchestra.run.smithi146.stdout: Cleanup : kernel-5.14.0-437.el9.x86_64 2/2 2024-05-04T12:00:24.341 INFO:teuthology.orchestra.run.smithi079.stdout:Is this ok [y/N]: 2024-05-04T12:00:24.342 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum reinstall -y kernel || true 2024-05-04T12:00:24.837 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:00:03 ago on Sat 04 May 2024 12:00:21 PM UTC. 2024-05-04T12:00:24.954 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-05-04T12:00:24.955 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-05-04T12:00:24.955 INFO:teuthology.orchestra.run.smithi079.stdout: Package Architecture Version Repository Size 2024-05-04T12:00:24.955 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-05-04T12:00:24.955 INFO:teuthology.orchestra.run.smithi079.stdout:Reinstalling: 2024-05-04T12:00:24.955 INFO:teuthology.orchestra.run.smithi079.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-05-04T12:00:24.955 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-05-04T12:00:24.955 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-05-04T12:00:24.955 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-05-04T12:00:24.955 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-05-04T12:00:24.955 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 255 k 2024-05-04T12:00:24.956 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 0 2024-05-04T12:00:24.956 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-05-04T12:00:25.031 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: kernel-5.14.0-437.el9.x86_64 2/2 2024-05-04T12:00:25.031 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 1/2 2024-05-04T12:00:25.342 INFO:teuthology.orchestra.run.smithi079.stdout:kernel-5.14.0-437.el9.x86_64.rpm 1.9 MB/s | 255 kB 00:00 2024-05-04T12:00:25.342 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-05-04T12:00:25.342 INFO:teuthology.orchestra.run.smithi079.stdout:Total 659 kB/s | 255 kB 00:00 2024-05-04T12:00:25.363 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-05-04T12:00:25.539 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-05-04T12:00:25.539 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-05-04T12:00:25.544 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-05-04T12:00:25.544 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-05-04T12:00:25.666 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-05-04T12:00:25.731 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 2/2 2024-05-04T12:00:25.732 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-05-04T12:00:25.732 INFO:teuthology.orchestra.run.smithi146.stdout:Reinstalled: 2024-05-04T12:00:25.732 INFO:teuthology.orchestra.run.smithi146.stdout: kernel-5.14.0-437.el9.x86_64 2024-05-04T12:00:25.732 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-05-04T12:00:25.732 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-05-04T12:00:25.751 INFO:teuthology.orchestra.run.smithi079.stdout: Reinstalling : kernel-5.14.0-437.el9.x86_64 1/2 2024-05-04T12:00:25.751 INFO:teuthology.orchestra.run.smithi079.stdout: Cleanup : kernel-5.14.0-437.el9.x86_64 2/2 2024-05-04T12:00:25.924 DEBUG:teuthology.orchestra.run.smithi146:> rpm -q kernel | sort -rV | head -n 1 2024-05-04T12:00:25.971 INFO:teuthology.orchestra.run.smithi146.stdout:kernel-5.14.0-437.el9.x86_64 2024-05-04T12:00:25.972 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-437.el9.x86_64 2024-05-04T12:00:25.972 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-05-04T12:00:25.972 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2024-05-04T12:00:25.972 DEBUG:teuthology.orchestra.run.smithi146:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-05-04T12:00:26.003 INFO:teuthology.orchestra.run.smithi146.stdout:ttyS1 2024-05-04T12:00:26.023 DEBUG:teuthology.parallel:result is None 2024-05-04T12:00:26.502 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: kernel-5.14.0-437.el9.x86_64 2/2 2024-05-04T12:00:26.502 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 1/2 2024-05-04T12:00:27.196 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 2/2 2024-05-04T12:00:27.197 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-05-04T12:00:27.197 INFO:teuthology.orchestra.run.smithi079.stdout:Reinstalled: 2024-05-04T12:00:27.197 INFO:teuthology.orchestra.run.smithi079.stdout: kernel-5.14.0-437.el9.x86_64 2024-05-04T12:00:27.197 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-05-04T12:00:27.197 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-05-04T12:00:27.388 DEBUG:teuthology.orchestra.run.smithi079:> rpm -q kernel | sort -rV | head -n 1 2024-05-04T12:00:27.435 INFO:teuthology.orchestra.run.smithi079.stdout:kernel-5.14.0-437.el9.x86_64 2024-05-04T12:00:27.435 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-437.el9.x86_64 2024-05-04T12:00:27.435 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-05-04T12:00:27.435 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2024-05-04T12:00:27.435 DEBUG:teuthology.orchestra.run.smithi079:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-05-04T12:00:27.461 INFO:teuthology.orchestra.run.smithi079.stdout:ttyS1 2024-05-04T12:00:27.483 DEBUG:teuthology.parallel:result is None 2024-05-04T12:00:27.483 INFO:teuthology.run_tasks:Running task internal.base... 2024-05-04T12:00:27.492 INFO:teuthology.task.internal:Creating test directory... 2024-05-04T12:00:27.492 DEBUG:teuthology.orchestra.run.smithi079:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-05-04T12:00:27.525 DEBUG:teuthology.orchestra.run.smithi146:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-05-04T12:00:27.543 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-05-04T12:00:27.555 INFO:teuthology.run_tasks:Running task internal.archive... 2024-05-04T12:00:27.568 INFO:teuthology.task.internal:Creating archive directory... 2024-05-04T12:00:27.568 DEBUG:teuthology.orchestra.run.smithi079:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-05-04T12:00:27.580 DEBUG:teuthology.orchestra.run.smithi146:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-05-04T12:00:27.610 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-05-04T12:00:27.621 INFO:teuthology.task.internal:Enabling coredump saving... 2024-05-04T12:00:27.621 DEBUG:teuthology.orchestra.run.smithi079:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-05-04T12:00:27.640 DEBUG:teuthology.orchestra.run.smithi146:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-05-04T12:00:27.693 INFO:teuthology.orchestra.run.smithi079.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-04T12:00:27.703 INFO:teuthology.orchestra.run.smithi079.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-04T12:00:27.729 INFO:teuthology.orchestra.run.smithi146.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-04T12:00:27.740 INFO:teuthology.orchestra.run.smithi146.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-04T12:00:27.742 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-05-04T12:00:27.753 INFO:teuthology.task.internal:Configuring sudo... 2024-05-04T12:00:27.753 DEBUG:teuthology.orchestra.run.smithi079:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-05-04T12:00:27.756 DEBUG:teuthology.orchestra.run.smithi146:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-05-04T12:00:27.783 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-05-04T12:00:27.795 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-05-04T12:00:27.795 DEBUG:teuthology.orchestra.run.smithi079:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-05-04T12:00:27.820 DEBUG:teuthology.orchestra.run.smithi146:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-05-04T12:00:27.839 DEBUG:teuthology.orchestra.run.smithi079:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-04T12:00:27.920 DEBUG:teuthology.orchestra.run.smithi079:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-04T12:00:27.963 DEBUG:teuthology.orchestra.run.smithi079:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-04T12:00:28.051 DEBUG:teuthology.orchestra.run.smithi079:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-04T12:00:28.083 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-05-04T12:00:28.084 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-05-04T12:00:28.146 DEBUG:teuthology.orchestra.run.smithi146:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-04T12:00:28.199 DEBUG:teuthology.orchestra.run.smithi146:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-04T12:00:28.238 DEBUG:teuthology.orchestra.run.smithi146:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-04T12:00:28.330 DEBUG:teuthology.orchestra.run.smithi146:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-04T12:00:28.363 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-05-04T12:00:28.364 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-05-04T12:00:28.430 DEBUG:teuthology.orchestra.run.smithi079:> sudo service rsyslog restart 2024-05-04T12:00:28.432 DEBUG:teuthology.orchestra.run.smithi146:> sudo service rsyslog restart 2024-05-04T12:00:28.469 INFO:teuthology.orchestra.run.smithi079.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-04T12:00:28.562 INFO:teuthology.orchestra.run.smithi146.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-04T12:00:28.981 INFO:teuthology.run_tasks:Running task internal.timer... 2024-05-04T12:00:28.990 INFO:teuthology.task.internal:Starting timer... 2024-05-04T12:00:28.990 INFO:teuthology.run_tasks:Running task pcp... 2024-05-04T12:00:29.006 INFO:teuthology.run_tasks:Running task selinux... 2024-05-04T12:00:29.019 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-05-04T12:00:29.019 DEBUG:teuthology.orchestra.run.smithi079:> sudo service auditd rotate 2024-05-04T12:00:29.080 INFO:teuthology.orchestra.run.smithi079.stdout:Rotating logs: 2024-05-04T12:00:29.082 DEBUG:teuthology.orchestra.run.smithi146:> sudo service auditd rotate 2024-05-04T12:00:29.143 INFO:teuthology.orchestra.run.smithi146.stdout:Rotating logs: 2024-05-04T12:00:29.145 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-05-04T12:00:29.145 DEBUG:teuthology.orchestra.run.smithi079:> /usr/sbin/getenforce 2024-05-04T12:00:29.162 INFO:teuthology.orchestra.run.smithi079.stdout:Permissive 2024-05-04T12:00:29.162 DEBUG:teuthology.orchestra.run.smithi146:> /usr/sbin/getenforce 2024-05-04T12:00:29.186 INFO:teuthology.orchestra.run.smithi146.stdout:Permissive 2024-05-04T12:00:29.187 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi079.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi146.front.sepia.ceph.com': 'permissive'} 2024-05-04T12:00:29.187 DEBUG:teuthology.orchestra.run.smithi079:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-04T12:00:29.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-04T12:00:29.226 DEBUG:teuthology.orchestra.run.smithi146:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-04T12:00:29.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-04T12:00:29.251 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-05-04T12:00:29.251 DEBUG:teuthology.orchestra.run.smithi079:> sudo /usr/sbin/setenforce permissive 2024-05-04T12:00:29.290 DEBUG:teuthology.orchestra.run.smithi146:> sudo /usr/sbin/setenforce permissive 2024-05-04T12:00:29.315 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-05-04T12:00:29.323 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-05-04T12:00:29.328 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-05-04T12:00:29.532 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-05-04T12:00:29.548 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-05-04T12:00:29.549 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi079.front.sepia.ceph.com,smithi146.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-05-04T12:04:29.558 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi079.front.sepia.ceph.com'), Remote(name='ubuntu@smithi146.front.sepia.ceph.com')] 2024-05-04T12:04:29.559 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi079.front.sepia.ceph.com' 2024-05-04T12:04:29.560 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi079.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-04T12:04:29.645 DEBUG:teuthology.orchestra.run.smithi079:> true 2024-05-04T12:04:29.738 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi079.front.sepia.ceph.com' 2024-05-04T12:04:29.738 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-05-04T12:04:29.739 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-04T12:04:29.819 DEBUG:teuthology.orchestra.run.smithi146:> true 2024-05-04T12:04:29.897 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-05-04T12:04:29.897 INFO:teuthology.run_tasks:Running task clock... 2024-05-04T12:04:29.907 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-05-04T12:04:29.907 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-05-04T12:04:29.908 DEBUG:teuthology.orchestra.run.smithi079:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-04T12:04:29.910 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-05-04T12:04:29.910 DEBUG:teuthology.orchestra.run.smithi146:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-04T12:04:29.938 INFO:teuthology.orchestra.run.smithi079.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-05-04T12:04:29.952 INFO:teuthology.orchestra.run.smithi079.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-05-04T12:04:29.965 INFO:teuthology.orchestra.run.smithi146.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-05-04T12:04:29.976 INFO:teuthology.orchestra.run.smithi079.stderr:sudo: ntpd: command not found 2024-05-04T12:04:29.978 INFO:teuthology.orchestra.run.smithi146.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-05-04T12:04:29.987 INFO:teuthology.orchestra.run.smithi079.stdout:506 Cannot talk to daemon 2024-05-04T12:04:30.000 INFO:teuthology.orchestra.run.smithi079.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-05-04T12:04:30.001 INFO:teuthology.orchestra.run.smithi146.stderr:sudo: ntpd: command not found 2024-05-04T12:04:30.011 INFO:teuthology.orchestra.run.smithi146.stdout:506 Cannot talk to daemon 2024-05-04T12:04:30.014 INFO:teuthology.orchestra.run.smithi079.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-05-04T12:04:30.024 INFO:teuthology.orchestra.run.smithi146.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-05-04T12:04:30.038 INFO:teuthology.orchestra.run.smithi146.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-05-04T12:04:30.068 INFO:teuthology.orchestra.run.smithi079.stderr:bash: line 1: ntpq: command not found 2024-05-04T12:04:30.071 INFO:teuthology.orchestra.run.smithi079.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-04T12:04:30.071 INFO:teuthology.orchestra.run.smithi079.stdout:=============================================================================== 2024-05-04T12:04:30.071 INFO:teuthology.orchestra.run.smithi079.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-04T12:04:30.071 INFO:teuthology.orchestra.run.smithi079.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-04T12:04:30.071 INFO:teuthology.orchestra.run.smithi079.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-04T12:04:30.071 INFO:teuthology.orchestra.run.smithi079.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-04T12:04:30.099 INFO:teuthology.orchestra.run.smithi146.stderr:bash: line 1: ntpq: command not found 2024-05-04T12:04:30.103 INFO:teuthology.orchestra.run.smithi146.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-04T12:04:30.103 INFO:teuthology.orchestra.run.smithi146.stdout:=============================================================================== 2024-05-04T12:04:30.103 INFO:teuthology.orchestra.run.smithi146.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-04T12:04:30.103 INFO:teuthology.orchestra.run.smithi146.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-04T12:04:30.103 INFO:teuthology.orchestra.run.smithi146.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-04T12:04:30.103 INFO:teuthology.orchestra.run.smithi146.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-04T12:04:30.103 INFO:teuthology.run_tasks:Running task install... 2024-05-04T12:04:30.111 DEBUG:teuthology.task.install:project ceph 2024-05-04T12:04:30.111 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '09dbd6bba52d19ba471c33bc15c008e6ad158ea6'}} 2024-05-04T12:04:30.112 DEBUG:teuthology.task.install:config {'exclude_packages': ['ceph-volume'], 'tag': 'v18.2.1', 'flavor': 'default', 'sha1': '09dbd6bba52d19ba471c33bc15c008e6ad158ea6'} 2024-05-04T12:04:30.112 INFO:teuthology.task.install:Using flavor: default 2024-05-04T12:04:30.117 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-05-04T12:04:30.117 INFO:teuthology.task.install:extra packages: [] 2024-05-04T12:04:30.118 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '09dbd6bba52d19ba471c33bc15c008e6ad158ea6', 'tag': 'v18.2.1', 'wait_for_package': False} 2024-05-04T12:04:30.121 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-05-04T12:04:30.121 INFO:teuthology.packaging:ref: None 2024-05-04T12:04:30.121 INFO:teuthology.packaging:tag: v18.2.1 2024-05-04T12:04:30.122 INFO:teuthology.packaging:branch: None 2024-05-04T12:04:30.122 INFO:teuthology.packaging:sha1: 09dbd6bba52d19ba471c33bc15c008e6ad158ea6 2024-05-04T12:04:30.325 DEBUG:teuthology.repo_utils:git ls-remote https://github.com/ceph/ceph-ci v18.2.1^{} -> 7fe91d5d5842e04be3b4f514d6dd990c54b29c76 2024-05-04T12:04:30.326 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=7fe91d5d5842e04be3b4f514d6dd990c54b29c76 2024-05-04T12:04:30.330 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '09dbd6bba52d19ba471c33bc15c008e6ad158ea6', 'tag': 'v18.2.1', 'wait_for_package': False} 2024-05-04T12:04:30.330 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-05-04T12:04:30.330 INFO:teuthology.packaging:ref: None 2024-05-04T12:04:30.331 INFO:teuthology.packaging:tag: v18.2.1 2024-05-04T12:04:30.331 INFO:teuthology.packaging:branch: None 2024-05-04T12:04:30.331 INFO:teuthology.packaging:sha1: 09dbd6bba52d19ba471c33bc15c008e6ad158ea6 2024-05-04T12:04:30.331 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=7fe91d5d5842e04be3b4f514d6dd990c54b29c76 2024-05-04T12:04:30.490 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/reef/7fe91d5d5842e04be3b4f514d6dd990c54b29c76/centos/9/flavors/default/ 2024-05-04T12:04:30.490 INFO:teuthology.task.install.rpm:Package version is 18.2.1-0 2024-05-04T12:04:30.532 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/reef/7fe91d5d5842e04be3b4f514d6dd990c54b29c76/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/reef/7fe91d5d5842e04be3b4f514d6dd990c54b29c76/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/reef/7fe91d5d5842e04be3b4f514d6dd990c54b29c76/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-05-04T12:04:30.532 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-05-04T12:04:30.533 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-05-04T12:04:30.558 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-05-04T12:04:30.558 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-05-04T12:04:30.558 INFO:teuthology.packaging:ref: None 2024-05-04T12:04:30.558 INFO:teuthology.packaging:tag: v18.2.1 2024-05-04T12:04:30.558 INFO:teuthology.packaging:branch: None 2024-05-04T12:04:30.558 INFO:teuthology.packaging:sha1: 09dbd6bba52d19ba471c33bc15c008e6ad158ea6 2024-05-04T12:04:30.558 DEBUG:teuthology.orchestra.run.smithi079:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/v18.2.1/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-05-04T12:04:30.567 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/reef/7fe91d5d5842e04be3b4f514d6dd990c54b29c76/centos/9/flavors/default/ 2024-05-04T12:04:30.567 INFO:teuthology.task.install.rpm:Package version is 18.2.1-0 2024-05-04T12:04:30.601 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/reef/7fe91d5d5842e04be3b4f514d6dd990c54b29c76/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/reef/7fe91d5d5842e04be3b4f514d6dd990c54b29c76/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/reef/7fe91d5d5842e04be3b4f514d6dd990c54b29c76/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-05-04T12:04:30.601 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-05-04T12:04:30.601 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-05-04T12:04:30.622 DEBUG:teuthology.orchestra.run.smithi079:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-05-04T12:04:30.628 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-05-04T12:04:30.628 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-05-04T12:04:30.628 INFO:teuthology.packaging:ref: None 2024-05-04T12:04:30.628 INFO:teuthology.packaging:tag: v18.2.1 2024-05-04T12:04:30.628 INFO:teuthology.packaging:branch: None 2024-05-04T12:04:30.628 INFO:teuthology.packaging:sha1: 09dbd6bba52d19ba471c33bc15c008e6ad158ea6 2024-05-04T12:04:30.628 DEBUG:teuthology.orchestra.run.smithi146:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/v18.2.1/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-05-04T12:04:30.693 DEBUG:teuthology.orchestra.run.smithi146:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-05-04T12:04:30.695 DEBUG:teuthology.orchestra.run.smithi079:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-05-04T12:04:30.758 INFO:teuthology.orchestra.run.smithi079.stdout:check_obsoletes = 1 2024-05-04T12:04:30.760 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum clean all 2024-05-04T12:04:30.768 DEBUG:teuthology.orchestra.run.smithi146:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-05-04T12:04:30.831 INFO:teuthology.orchestra.run.smithi146.stdout:check_obsoletes = 1 2024-05-04T12:04:30.832 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum clean all 2024-05-04T12:04:30.985 INFO:teuthology.orchestra.run.smithi079.stdout:50 files removed 2024-05-04T12:04:31.012 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install ceph-radosgw 2024-05-04T12:04:31.060 INFO:teuthology.orchestra.run.smithi146.stdout:50 files removed 2024-05-04T12:04:31.087 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install ceph-radosgw 2024-05-04T12:04:31.466 INFO:teuthology.orchestra.run.smithi079.stdout:ceph packages for x86_64 392 kB/s | 76 kB 00:00 2024-05-04T12:04:31.468 INFO:teuthology.orchestra.run.smithi146.stdout:ceph packages for x86_64 624 kB/s | 76 kB 00:00 2024-05-04T12:04:31.652 INFO:teuthology.orchestra.run.smithi146.stdout:ceph noarch packages 67 kB/s | 9.4 kB 00:00 2024-05-04T12:04:31.667 INFO:teuthology.orchestra.run.smithi079.stdout:ceph noarch packages 60 kB/s | 9.4 kB 00:00 2024-05-04T12:04:31.845 INFO:teuthology.orchestra.run.smithi146.stdout:ceph source packages 14 kB/s | 2.2 kB 00:00 2024-05-04T12:04:31.884 INFO:teuthology.orchestra.run.smithi079.stdout:ceph source packages 12 kB/s | 2.2 kB 00:00 2024-05-04T12:04:32.545 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.0 MB 00:00 2024-05-04T12:04:32.828 INFO:teuthology.orchestra.run.smithi079.stdout:CentOS Stream 9 - BaseOS 8.8 MB/s | 8.0 MB 00:00 2024-05-04T12:04:34.254 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS Stream 9 - AppStream 32 MB/s | 19 MB 00:00 2024-05-04T12:04:37.585 INFO:teuthology.orchestra.run.smithi079.stdout:CentOS Stream 9 - AppStream 5.1 MB/s | 19 MB 00:03 2024-05-04T12:04:38.589 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS Stream 9 - CRB 8.3 MB/s | 6.2 MB 00:00 2024-05-04T12:04:39.998 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS Stream 9 - Extras packages 67 kB/s | 16 kB 00:00 2024-05-04T12:04:41.494 INFO:teuthology.orchestra.run.smithi146.stdout:Extra Packages for Enterprise Linux 15 MB/s | 21 MB 00:01 2024-05-04T12:04:43.010 INFO:teuthology.orchestra.run.smithi079.stdout:CentOS Stream 9 - CRB 3.3 MB/s | 6.2 MB 00:01 2024-05-04T12:04:44.595 INFO:teuthology.orchestra.run.smithi079.stdout:CentOS Stream 9 - Extras packages 36 kB/s | 16 kB 00:00 2024-05-04T12:04:46.641 INFO:teuthology.orchestra.run.smithi146.stdout:lab-extras 16 kB/s | 1.7 kB 00:00 2024-05-04T12:04:48.766 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-05-04T12:04:48.768 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-05-04T12:04:48.768 INFO:teuthology.orchestra.run.smithi146.stdout: Package Arch Version Repository Size 2024-05-04T12:04:48.768 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-05-04T12:04:48.769 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2024-05-04T12:04:48.769 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-radosgw x86_64 2:18.2.1-0.el9 ceph 7.7 M 2024-05-04T12:04:48.769 INFO:teuthology.orchestra.run.smithi146.stdout:Upgrading: 2024-05-04T12:04:48.769 INFO:teuthology.orchestra.run.smithi146.stdout: librados2 x86_64 2:18.2.1-0.el9 ceph 3.3 M 2024-05-04T12:04:48.769 INFO:teuthology.orchestra.run.smithi146.stdout: librbd1 x86_64 2:18.2.1-0.el9 ceph 3.0 M 2024-05-04T12:04:48.769 INFO:teuthology.orchestra.run.smithi146.stdout:Installing dependencies: 2024-05-04T12:04:48.770 INFO:teuthology.orchestra.run.smithi146.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-05-04T12:04:48.770 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-base x86_64 2:18.2.1-0.el9 ceph 5.2 M 2024-05-04T12:04:48.770 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-common x86_64 2:18.2.1-0.el9 ceph 18 M 2024-05-04T12:04:48.770 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-selinux x86_64 2:18.2.1-0.el9 ceph 24 k 2024-05-04T12:04:48.770 INFO:teuthology.orchestra.run.smithi146.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2024-05-04T12:04:48.770 INFO:teuthology.orchestra.run.smithi146.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-05-04T12:04:48.771 INFO:teuthology.orchestra.run.smithi146.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-05-04T12:04:48.771 INFO:teuthology.orchestra.run.smithi146.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-05-04T12:04:48.771 INFO:teuthology.orchestra.run.smithi146.stdout: libcephfs2 x86_64 2:18.2.1-0.el9 ceph 658 k 2024-05-04T12:04:48.771 INFO:teuthology.orchestra.run.smithi146.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-05-04T12:04:48.771 INFO:teuthology.orchestra.run.smithi146.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-05-04T12:04:48.771 INFO:teuthology.orchestra.run.smithi146.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-05-04T12:04:48.771 INFO:teuthology.orchestra.run.smithi146.stdout: libradosstriper1 x86_64 2:18.2.1-0.el9 ceph 474 k 2024-05-04T12:04:48.771 INFO:teuthology.orchestra.run.smithi146.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-05-04T12:04:48.771 INFO:teuthology.orchestra.run.smithi146.stdout: librgw2 x86_64 2:18.2.1-0.el9 ceph 4.5 M 2024-05-04T12:04:48.772 INFO:teuthology.orchestra.run.smithi146.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-05-04T12:04:48.772 INFO:teuthology.orchestra.run.smithi146.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-05-04T12:04:48.772 INFO:teuthology.orchestra.run.smithi146.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-05-04T12:04:48.772 INFO:teuthology.orchestra.run.smithi146.stdout: python3-ceph-argparse x86_64 2:18.2.1-0.el9 ceph 45 k 2024-05-04T12:04:48.772 INFO:teuthology.orchestra.run.smithi146.stdout: python3-ceph-common x86_64 2:18.2.1-0.el9 ceph 124 k 2024-05-04T12:04:48.772 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cephfs x86_64 2:18.2.1-0.el9 ceph 161 k 2024-05-04T12:04:48.772 INFO:teuthology.orchestra.run.smithi146.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-05-04T12:04:48.772 INFO:teuthology.orchestra.run.smithi146.stdout: python3-rados x86_64 2:18.2.1-0.el9 ceph 321 k 2024-05-04T12:04:48.772 INFO:teuthology.orchestra.run.smithi146.stdout: python3-rbd x86_64 2:18.2.1-0.el9 ceph 297 k 2024-05-04T12:04:48.773 INFO:teuthology.orchestra.run.smithi146.stdout: python3-rgw x86_64 2:18.2.1-0.el9 ceph 99 k 2024-05-04T12:04:48.773 INFO:teuthology.orchestra.run.smithi146.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-05-04T12:04:48.773 INFO:teuthology.orchestra.run.smithi146.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-05-04T12:04:48.773 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-05-04T12:04:48.773 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-05-04T12:04:48.773 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-05-04T12:04:48.773 INFO:teuthology.orchestra.run.smithi146.stdout:Install 28 Packages 2024-05-04T12:04:48.773 INFO:teuthology.orchestra.run.smithi146.stdout:Upgrade 2 Packages 2024-05-04T12:04:48.774 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-05-04T12:04:48.774 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 53 M 2024-05-04T12:04:48.775 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-05-04T12:04:49.217 INFO:teuthology.orchestra.run.smithi146.stdout:(1/30): ceph-base-18.2.1-0.el9.x86_64.rpm 20 MB/s | 5.2 MB 00:00 2024-05-04T12:04:49.493 INFO:teuthology.orchestra.run.smithi146.stdout:(2/30): ceph-radosgw-18.2.1-0.el9.x86_64.rpm 14 MB/s | 7.7 MB 00:00 2024-05-04T12:04:49.518 INFO:teuthology.orchestra.run.smithi146.stdout:(3/30): ceph-selinux-18.2.1-0.el9.x86_64.rpm 80 kB/s | 24 kB 00:00 2024-05-04T12:04:49.551 INFO:teuthology.orchestra.run.smithi146.stdout:(4/30): libcephfs2-18.2.1-0.el9.x86_64.rpm 11 MB/s | 658 kB 00:00 2024-05-04T12:04:50.052 INFO:teuthology.orchestra.run.smithi146.stdout:(5/30): ceph-common-18.2.1-0.el9.x86_64.rpm 17 MB/s | 18 MB 00:01 2024-05-04T12:04:50.094 INFO:teuthology.orchestra.run.smithi146.stdout:(6/30): libradosstriper1-18.2.1-0.el9.x86_64.rp 824 kB/s | 474 kB 00:00 2024-05-04T12:04:50.110 INFO:teuthology.orchestra.run.smithi146.stdout:(7/30): python3-ceph-argparse-18.2.1-0.el9.x86_ 771 kB/s | 45 kB 00:00 2024-05-04T12:04:50.144 INFO:teuthology.orchestra.run.smithi146.stdout:(8/30): python3-ceph-common-18.2.1-0.el9.x86_64 2.4 MB/s | 124 kB 00:00 2024-05-04T12:04:50.203 INFO:teuthology.orchestra.run.smithi146.stdout:(9/30): librgw2-18.2.1-0.el9.x86_64.rpm 6.8 MB/s | 4.5 MB 00:00 2024-05-04T12:04:50.228 INFO:teuthology.orchestra.run.smithi146.stdout:(10/30): python3-cephfs-18.2.1-0.el9.x86_64.rpm 1.3 MB/s | 161 kB 00:00 2024-05-04T12:04:50.253 INFO:teuthology.orchestra.run.smithi146.stdout:(11/30): python3-rados-18.2.1-0.el9.x86_64.rpm 2.9 MB/s | 321 kB 00:00 2024-05-04T12:04:50.287 INFO:teuthology.orchestra.run.smithi146.stdout:(12/30): python3-rbd-18.2.1-0.el9.x86_64.rpm 3.5 MB/s | 297 kB 00:00 2024-05-04T12:04:50.312 INFO:teuthology.orchestra.run.smithi146.stdout:(13/30): python3-rgw-18.2.1-0.el9.x86_64.rpm 1.2 MB/s | 99 kB 00:00 2024-05-04T12:04:50.396 INFO:teuthology.orchestra.run.smithi146.stdout:(14/30): boost-program-options-1.75.0-8.el9.x86 752 kB/s | 107 kB 00:00 2024-05-04T12:04:50.421 INFO:teuthology.orchestra.run.smithi146.stdout:(15/30): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.2 MB/s | 160 kB 00:00 2024-05-04T12:04:50.446 INFO:teuthology.orchestra.run.smithi146.stdout:(16/30): librabbitmq-0.11.0-7.el9.x86_64.rpm 338 kB/s | 45 kB 00:00 2024-05-04T12:04:50.521 INFO:teuthology.orchestra.run.smithi146.stdout:(17/30): librdkafka-1.6.1-102.el9.x86_64.rpm 5.2 MB/s | 662 kB 00:00 2024-05-04T12:04:50.547 INFO:teuthology.orchestra.run.smithi146.stdout:(18/30): python3-prettytable-0.7.2-27.el9.noarc 422 kB/s | 42 kB 00:00 2024-05-04T12:04:50.572 INFO:teuthology.orchestra.run.smithi146.stdout:(19/30): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.3 MB/s | 292 kB 00:00 2024-05-04T12:04:50.905 INFO:teuthology.orchestra.run.smithi146.stdout:(20/30): fmt-8.1.1-5.el9.x86_64.rpm 288 kB/s | 111 kB 00:00 2024-05-04T12:04:50.939 INFO:teuthology.orchestra.run.smithi146.stdout:(21/30): gperftools-libs-2.9.1-2.el9.x86_64.rpm 788 kB/s | 309 kB 00:00 2024-05-04T12:04:50.981 INFO:teuthology.orchestra.run.smithi146.stdout:(22/30): libarrow-doc-9.0.0-11.el9.noarch.rpm 345 kB/s | 26 kB 00:00 2024-05-04T12:04:51.015 INFO:teuthology.orchestra.run.smithi146.stdout:(23/30): liboath-2.6.7-2.el9.x86_64.rpm 644 kB/s | 49 kB 00:00 2024-05-04T12:04:51.073 INFO:teuthology.orchestra.run.smithi146.stdout:(24/30): libunwind-1.6.2-1.el9.x86_64.rpm 733 kB/s | 67 kB 00:00 2024-05-04T12:04:51.182 INFO:teuthology.orchestra.run.smithi146.stdout:(25/30): parquet-libs-9.0.0-11.el9.x86_64.rpm 4.9 MB/s | 839 kB 00:00 2024-05-04T12:04:51.198 INFO:teuthology.orchestra.run.smithi146.stdout:(26/30): re2-20211101-3.el9.x86_64.rpm 1.5 MB/s | 192 kB 00:00 2024-05-04T12:04:51.266 INFO:teuthology.orchestra.run.smithi146.stdout:(27/30): libarrow-9.0.0-11.el9.x86_64.rpm 6.4 MB/s | 4.4 MB 00:00 2024-05-04T12:04:51.341 INFO:teuthology.orchestra.run.smithi146.stdout:(28/30): thrift-0.15.0-2.el9.x86_64.rpm 9.9 MB/s | 1.6 MB 00:00 2024-05-04T12:04:51.442 INFO:teuthology.orchestra.run.smithi146.stdout:(29/30): librados2-18.2.1-0.el9.x86_64.rpm 13 MB/s | 3.3 MB 00:00 2024-05-04T12:04:51.483 INFO:teuthology.orchestra.run.smithi146.stdout:(30/30): librbd1-18.2.1-0.el9.x86_64.rpm 14 MB/s | 3.0 MB 00:00 2024-05-04T12:04:51.488 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-05-04T12:04:51.488 INFO:teuthology.orchestra.run.smithi146.stdout:Total 19 MB/s | 53 MB 00:02 2024-05-04T12:04:51.556 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-05-04T12:04:51.591 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-05-04T12:04:51.591 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-05-04T12:04:52.013 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-05-04T12:04:52.019 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-05-04T12:04:53.405 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-05-04T12:04:53.494 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/32 2024-05-04T12:04:53.530 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/32 2024-05-04T12:04:53.579 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/32 2024-05-04T12:04:53.892 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/32 2024-05-04T12:04:54.004 INFO:teuthology.orchestra.run.smithi146.stdout: Upgrading : librados2-2:18.2.1-0.el9.x86_64 5/32 2024-05-04T12:04:54.097 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: librados2-2:18.2.1-0.el9.x86_64 5/32 2024-05-04T12:04:54.186 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-rados-2:18.2.1-0.el9.x86_64 6/32 2024-05-04T12:04:54.222 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libcephfs2-2:18.2.1-0.el9.x86_64 7/32 2024-05-04T12:04:54.285 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: libcephfs2-2:18.2.1-0.el9.x86_64 7/32 2024-05-04T12:04:54.314 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/32 2024-05-04T12:04:54.357 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-ceph-argparse-2:18.2.1-0.el9.x86_64 9/32 2024-05-04T12:04:54.423 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-cephfs-2:18.2.1-0.el9.x86_64 10/32 2024-05-04T12:04:54.450 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libradosstriper1-2:18.2.1-0.el9.x86_64 11/32 2024-05-04T12:04:54.515 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: libradosstriper1-2:18.2.1-0.el9.x86_64 11/32 2024-05-04T12:04:54.622 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/32 2024-05-04T12:04:54.657 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 13/32 2024-05-04T12:04:54.697 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/32 2024-05-04T12:04:54.722 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/32 2024-05-04T12:04:54.757 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 16/32 2024-05-04T12:04:54.793 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : fmt-8.1.1-5.el9.x86_64 17/32 2024-05-04T12:04:54.826 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 18/32 2024-05-04T12:04:55.126 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 19/32 2024-05-04T12:04:55.164 INFO:teuthology.orchestra.run.smithi146.stdout: Upgrading : librbd1-2:18.2.1-0.el9.x86_64 20/32 2024-05-04T12:04:55.254 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: librbd1-2:18.2.1-0.el9.x86_64 20/32 2024-05-04T12:04:55.285 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-rbd-2:18.2.1-0.el9.x86_64 21/32 2024-05-04T12:04:55.321 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 22/32 2024-05-04T12:04:55.726 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 23/32 2024-05-04T12:04:55.758 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : librgw2-2:18.2.1-0.el9.x86_64 24/32 2024-05-04T12:04:55.829 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: librgw2-2:18.2.1-0.el9.x86_64 24/32 2024-05-04T12:04:55.907 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-rgw-2:18.2.1-0.el9.x86_64 25/32 2024-05-04T12:04:55.936 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-ceph-common-2:18.2.1-0.el9.x86_64 26/32 2024-05-04T12:04:58.023 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-common-2:18.2.1-0.el9.x86_64 27/32 2024-05-04T12:04:58.065 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-common-2:18.2.1-0.el9.x86_64 27/32 2024-05-04T12:04:58.605 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-common-2:18.2.1-0.el9.x86_64 27/32 2024-05-04T12:04:58.652 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-base-2:18.2.1-0.el9.x86_64 28/32 2024-05-04T12:04:58.746 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-base-2:18.2.1-0.el9.x86_64 28/32 2024-05-04T12:04:58.746 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-05-04T12:04:58.746 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-05-04T12:04:58.746 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-05-04T12:04:58.778 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-selinux-2:18.2.1-0.el9.x86_64 29/32 2024-05-04T12:05:08.259 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-selinux-2:18.2.1-0.el9.x86_64 29/32 2024-05-04T12:05:08.260 INFO:teuthology.orchestra.run.smithi146.stdout:skipping the directory /sys 2024-05-04T12:05:08.260 INFO:teuthology.orchestra.run.smithi146.stdout:skipping the directory /proc 2024-05-04T12:05:08.260 INFO:teuthology.orchestra.run.smithi146.stdout:skipping the directory /mnt 2024-05-04T12:05:08.260 INFO:teuthology.orchestra.run.smithi146.stdout:skipping the directory /var/tmp 2024-05-04T12:05:08.260 INFO:teuthology.orchestra.run.smithi146.stdout:skipping the directory /home 2024-05-04T12:05:08.260 INFO:teuthology.orchestra.run.smithi146.stdout:skipping the directory /root 2024-05-04T12:05:08.260 INFO:teuthology.orchestra.run.smithi146.stdout:skipping the directory /tmp 2024-05-04T12:05:08.260 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-05-04T12:05:08.987 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-radosgw-2:18.2.1-0.el9.x86_64 30/32 2024-05-04T12:05:09.012 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-radosgw-2:18.2.1-0.el9.x86_64 30/32 2024-05-04T12:05:09.012 INFO:teuthology.orchestra.run.smithi146.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-04T12:05:09.012 INFO:teuthology.orchestra.run.smithi146.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-05-04T12:05:09.012 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-04T12:05:09.012 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-04T12:05:09.012 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-05-04T12:05:09.014 INFO:teuthology.orchestra.run.smithi146.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-05-04T12:05:09.105 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-05-04T12:05:09.106 INFO:teuthology.orchestra.run.smithi146.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 32/32 2024-05-04T12:05:09.697 INFO:teuthology.orchestra.run.smithi079.stdout:Extra Packages for Enterprise Linux 861 kB/s | 21 MB 00:24 2024-05-04T12:05:10.242 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 32/32 2024-05-04T12:05:10.242 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-base-2:18.2.1-0.el9.x86_64 1/32 2024-05-04T12:05:10.242 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-common-2:18.2.1-0.el9.x86_64 2/32 2024-05-04T12:05:10.242 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-radosgw-2:18.2.1-0.el9.x86_64 3/32 2024-05-04T12:05:10.242 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-selinux-2:18.2.1-0.el9.x86_64 4/32 2024-05-04T12:05:10.243 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libcephfs2-2:18.2.1-0.el9.x86_64 5/32 2024-05-04T12:05:10.243 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libradosstriper1-2:18.2.1-0.el9.x86_64 6/32 2024-05-04T12:05:10.243 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : librgw2-2:18.2.1-0.el9.x86_64 7/32 2024-05-04T12:05:10.243 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-ceph-argparse-2:18.2.1-0.el9.x86_64 8/32 2024-05-04T12:05:10.243 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-ceph-common-2:18.2.1-0.el9.x86_64 9/32 2024-05-04T12:05:10.243 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-cephfs-2:18.2.1-0.el9.x86_64 10/32 2024-05-04T12:05:10.243 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-rados-2:18.2.1-0.el9.x86_64 11/32 2024-05-04T12:05:10.243 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-rbd-2:18.2.1-0.el9.x86_64 12/32 2024-05-04T12:05:10.243 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-rgw-2:18.2.1-0.el9.x86_64 13/32 2024-05-04T12:05:10.243 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/32 2024-05-04T12:05:10.243 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/32 2024-05-04T12:05:10.243 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/32 2024-05-04T12:05:10.243 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/32 2024-05-04T12:05:10.243 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/32 2024-05-04T12:05:10.244 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/32 2024-05-04T12:05:10.244 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 20/32 2024-05-04T12:05:10.244 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 21/32 2024-05-04T12:05:10.244 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 22/32 2024-05-04T12:05:10.244 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 23/32 2024-05-04T12:05:10.244 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 24/32 2024-05-04T12:05:10.244 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 25/32 2024-05-04T12:05:10.244 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 26/32 2024-05-04T12:05:10.244 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : re2-1:20211101-3.el9.x86_64 27/32 2024-05-04T12:05:10.244 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 28/32 2024-05-04T12:05:10.244 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : librados2-2:18.2.1-0.el9.x86_64 29/32 2024-05-04T12:05:10.244 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 30/32 2024-05-04T12:05:10.244 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : librbd1-2:18.2.1-0.el9.x86_64 31/32 2024-05-04T12:05:10.985 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 32/32 2024-05-04T12:05:10.985 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-05-04T12:05:10.985 INFO:teuthology.orchestra.run.smithi146.stdout:Upgraded: 2024-05-04T12:05:10.985 INFO:teuthology.orchestra.run.smithi146.stdout: librados2-2:18.2.1-0.el9.x86_64 librbd1-2:18.2.1-0.el9.x86_64 2024-05-04T12:05:10.985 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2024-05-04T12:05:10.986 INFO:teuthology.orchestra.run.smithi146.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-05-04T12:05:10.986 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-base-2:18.2.1-0.el9.x86_64 2024-05-04T12:05:10.986 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-common-2:18.2.1-0.el9.x86_64 2024-05-04T12:05:10.986 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-radosgw-2:18.2.1-0.el9.x86_64 2024-05-04T12:05:10.986 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-selinux-2:18.2.1-0.el9.x86_64 2024-05-04T12:05:10.986 INFO:teuthology.orchestra.run.smithi146.stdout: fmt-8.1.1-5.el9.x86_64 2024-05-04T12:05:10.986 INFO:teuthology.orchestra.run.smithi146.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-05-04T12:05:10.986 INFO:teuthology.orchestra.run.smithi146.stdout: libarrow-9.0.0-11.el9.x86_64 2024-05-04T12:05:10.986 INFO:teuthology.orchestra.run.smithi146.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-05-04T12:05:10.986 INFO:teuthology.orchestra.run.smithi146.stdout: libcephfs2-2:18.2.1-0.el9.x86_64 2024-05-04T12:05:10.986 INFO:teuthology.orchestra.run.smithi146.stdout: liboath-2.6.7-2.el9.x86_64 2024-05-04T12:05:10.986 INFO:teuthology.orchestra.run.smithi146.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-05-04T12:05:10.986 INFO:teuthology.orchestra.run.smithi146.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-05-04T12:05:10.987 INFO:teuthology.orchestra.run.smithi146.stdout: libradosstriper1-2:18.2.1-0.el9.x86_64 2024-05-04T12:05:10.987 INFO:teuthology.orchestra.run.smithi146.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-05-04T12:05:10.987 INFO:teuthology.orchestra.run.smithi146.stdout: librgw2-2:18.2.1-0.el9.x86_64 2024-05-04T12:05:10.987 INFO:teuthology.orchestra.run.smithi146.stdout: libunwind-1.6.2-1.el9.x86_64 2024-05-04T12:05:10.987 INFO:teuthology.orchestra.run.smithi146.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-05-04T12:05:10.987 INFO:teuthology.orchestra.run.smithi146.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-05-04T12:05:10.987 INFO:teuthology.orchestra.run.smithi146.stdout: python3-ceph-argparse-2:18.2.1-0.el9.x86_64 2024-05-04T12:05:10.987 INFO:teuthology.orchestra.run.smithi146.stdout: python3-ceph-common-2:18.2.1-0.el9.x86_64 2024-05-04T12:05:10.987 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cephfs-2:18.2.1-0.el9.x86_64 2024-05-04T12:05:10.987 INFO:teuthology.orchestra.run.smithi146.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-05-04T12:05:10.987 INFO:teuthology.orchestra.run.smithi146.stdout: python3-rados-2:18.2.1-0.el9.x86_64 2024-05-04T12:05:10.987 INFO:teuthology.orchestra.run.smithi146.stdout: python3-rbd-2:18.2.1-0.el9.x86_64 2024-05-04T12:05:10.987 INFO:teuthology.orchestra.run.smithi146.stdout: python3-rgw-2:18.2.1-0.el9.x86_64 2024-05-04T12:05:10.987 INFO:teuthology.orchestra.run.smithi146.stdout: re2-1:20211101-3.el9.x86_64 2024-05-04T12:05:10.987 INFO:teuthology.orchestra.run.smithi146.stdout: thrift-0.15.0-2.el9.x86_64 2024-05-04T12:05:10.988 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-05-04T12:05:10.988 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-05-04T12:05:11.267 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install ceph-test 2024-05-04T12:05:11.884 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:00:25 ago on Sat 04 May 2024 12:04:46 PM UTC. 2024-05-04T12:05:12.012 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-05-04T12:05:12.013 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-05-04T12:05:12.013 INFO:teuthology.orchestra.run.smithi146.stdout: Package Architecture Version Repository Size 2024-05-04T12:05:12.013 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-05-04T12:05:12.013 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2024-05-04T12:05:12.013 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-test x86_64 2:18.2.1-0.el9 ceph 40 M 2024-05-04T12:05:12.013 INFO:teuthology.orchestra.run.smithi146.stdout:Installing dependencies: 2024-05-04T12:05:12.013 INFO:teuthology.orchestra.run.smithi146.stdout: libcephsqlite x86_64 2:18.2.1-0.el9 ceph 165 k 2024-05-04T12:05:12.014 INFO:teuthology.orchestra.run.smithi146.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-05-04T12:05:12.014 INFO:teuthology.orchestra.run.smithi146.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-05-04T12:05:12.014 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-05-04T12:05:12.014 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-05-04T12:05:12.014 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-05-04T12:05:12.014 INFO:teuthology.orchestra.run.smithi146.stdout:Install 4 Packages 2024-05-04T12:05:12.014 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-05-04T12:05:12.015 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 40 M 2024-05-04T12:05:12.015 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 166 M 2024-05-04T12:05:12.015 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-05-04T12:05:12.312 INFO:teuthology.orchestra.run.smithi146.stdout:(1/4): socat-1.7.4.1-5.el9.x86_64.rpm 1.2 MB/s | 305 kB 00:00 2024-05-04T12:05:12.371 INFO:teuthology.orchestra.run.smithi146.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.1 MB/s | 64 kB 00:00 2024-05-04T12:05:12.984 INFO:teuthology.orchestra.run.smithi146.stdout:(3/4): libcephsqlite-18.2.1-0.el9.x86_64.rpm 178 kB/s | 165 kB 00:00 2024-05-04T12:05:13.394 INFO:teuthology.orchestra.run.smithi146.stdout:(4/4): ceph-test-18.2.1-0.el9.x86_64.rpm 30 MB/s | 40 MB 00:01 2024-05-04T12:05:13.396 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-05-04T12:05:13.397 INFO:teuthology.orchestra.run.smithi146.stdout:Total 29 MB/s | 40 MB 00:01 2024-05-04T12:05:13.423 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-05-04T12:05:13.443 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-05-04T12:05:13.443 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-05-04T12:05:13.773 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-05-04T12:05:13.773 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-05-04T12:05:14.277 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-05-04T12:05:14.383 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-05-04T12:05:14.432 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/4 2024-05-04T12:05:14.459 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libcephsqlite-2:18.2.1-0.el9.x86_64 3/4 2024-05-04T12:05:14.757 INFO:teuthology.orchestra.run.smithi079.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2024-05-04T12:05:16.865 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-05-04T12:05:16.867 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-05-04T12:05:16.867 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repository Size 2024-05-04T12:05:16.867 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-05-04T12:05:16.867 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-05-04T12:05:16.867 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-radosgw x86_64 2:18.2.1-0.el9 ceph 7.7 M 2024-05-04T12:05:16.867 INFO:teuthology.orchestra.run.smithi079.stdout:Upgrading: 2024-05-04T12:05:16.867 INFO:teuthology.orchestra.run.smithi079.stdout: librados2 x86_64 2:18.2.1-0.el9 ceph 3.3 M 2024-05-04T12:05:16.867 INFO:teuthology.orchestra.run.smithi079.stdout: librbd1 x86_64 2:18.2.1-0.el9 ceph 3.0 M 2024-05-04T12:05:16.868 INFO:teuthology.orchestra.run.smithi079.stdout:Installing dependencies: 2024-05-04T12:05:16.868 INFO:teuthology.orchestra.run.smithi079.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-05-04T12:05:16.868 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-base x86_64 2:18.2.1-0.el9 ceph 5.2 M 2024-05-04T12:05:16.868 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-common x86_64 2:18.2.1-0.el9 ceph 18 M 2024-05-04T12:05:16.868 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-selinux x86_64 2:18.2.1-0.el9 ceph 24 k 2024-05-04T12:05:16.868 INFO:teuthology.orchestra.run.smithi079.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2024-05-04T12:05:16.868 INFO:teuthology.orchestra.run.smithi079.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-05-04T12:05:16.868 INFO:teuthology.orchestra.run.smithi079.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-05-04T12:05:16.868 INFO:teuthology.orchestra.run.smithi079.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-05-04T12:05:16.868 INFO:teuthology.orchestra.run.smithi079.stdout: libcephfs2 x86_64 2:18.2.1-0.el9 ceph 658 k 2024-05-04T12:05:16.868 INFO:teuthology.orchestra.run.smithi079.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-05-04T12:05:16.868 INFO:teuthology.orchestra.run.smithi079.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-05-04T12:05:16.868 INFO:teuthology.orchestra.run.smithi079.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-05-04T12:05:16.868 INFO:teuthology.orchestra.run.smithi079.stdout: libradosstriper1 x86_64 2:18.2.1-0.el9 ceph 474 k 2024-05-04T12:05:16.869 INFO:teuthology.orchestra.run.smithi079.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-05-04T12:05:16.869 INFO:teuthology.orchestra.run.smithi079.stdout: librgw2 x86_64 2:18.2.1-0.el9 ceph 4.5 M 2024-05-04T12:05:16.869 INFO:teuthology.orchestra.run.smithi079.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-05-04T12:05:16.869 INFO:teuthology.orchestra.run.smithi079.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-05-04T12:05:16.869 INFO:teuthology.orchestra.run.smithi079.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-05-04T12:05:16.869 INFO:teuthology.orchestra.run.smithi079.stdout: python3-ceph-argparse x86_64 2:18.2.1-0.el9 ceph 45 k 2024-05-04T12:05:16.869 INFO:teuthology.orchestra.run.smithi079.stdout: python3-ceph-common x86_64 2:18.2.1-0.el9 ceph 124 k 2024-05-04T12:05:16.869 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cephfs x86_64 2:18.2.1-0.el9 ceph 161 k 2024-05-04T12:05:16.869 INFO:teuthology.orchestra.run.smithi079.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-05-04T12:05:16.869 INFO:teuthology.orchestra.run.smithi079.stdout: python3-rados x86_64 2:18.2.1-0.el9 ceph 321 k 2024-05-04T12:05:16.869 INFO:teuthology.orchestra.run.smithi079.stdout: python3-rbd x86_64 2:18.2.1-0.el9 ceph 297 k 2024-05-04T12:05:16.869 INFO:teuthology.orchestra.run.smithi079.stdout: python3-rgw x86_64 2:18.2.1-0.el9 ceph 99 k 2024-05-04T12:05:16.870 INFO:teuthology.orchestra.run.smithi079.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-05-04T12:05:16.870 INFO:teuthology.orchestra.run.smithi079.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-05-04T12:05:16.870 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-05-04T12:05:16.870 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-05-04T12:05:16.870 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-05-04T12:05:16.870 INFO:teuthology.orchestra.run.smithi079.stdout:Install 28 Packages 2024-05-04T12:05:16.870 INFO:teuthology.orchestra.run.smithi079.stdout:Upgrade 2 Packages 2024-05-04T12:05:16.870 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-05-04T12:05:16.870 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 53 M 2024-05-04T12:05:16.871 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-05-04T12:05:17.333 INFO:teuthology.orchestra.run.smithi079.stdout:(1/30): ceph-base-18.2.1-0.el9.x86_64.rpm 26 MB/s | 5.2 MB 00:00 2024-05-04T12:05:17.358 INFO:teuthology.orchestra.run.smithi079.stdout:(2/30): ceph-selinux-18.2.1-0.el9.x86_64.rpm 965 kB/s | 24 kB 00:00 2024-05-04T12:05:17.392 INFO:teuthology.orchestra.run.smithi079.stdout:(3/30): libcephfs2-18.2.1-0.el9.x86_64.rpm 19 MB/s | 658 kB 00:00 2024-05-04T12:05:17.426 INFO:teuthology.orchestra.run.smithi079.stdout:(4/30): libradosstriper1-18.2.1-0.el9.x86_64.rp 14 MB/s | 474 kB 00:00 2024-05-04T12:05:17.518 INFO:teuthology.orchestra.run.smithi079.stdout:(5/30): ceph-radosgw-18.2.1-0.el9.x86_64.rpm 20 MB/s | 7.7 MB 00:00 2024-05-04T12:05:17.543 INFO:teuthology.orchestra.run.smithi079.stdout:(6/30): python3-ceph-argparse-18.2.1-0.el9.x86_ 1.8 MB/s | 45 kB 00:00 2024-05-04T12:05:17.568 INFO:teuthology.orchestra.run.smithi079.stdout:(7/30): python3-ceph-common-18.2.1-0.el9.x86_64 4.9 MB/s | 124 kB 00:00 2024-05-04T12:05:17.594 INFO:teuthology.orchestra.run.smithi079.stdout:(8/30): python3-cephfs-18.2.1-0.el9.x86_64.rpm 6.3 MB/s | 161 kB 00:00 2024-05-04T12:05:17.619 INFO:teuthology.orchestra.run.smithi079.stdout:(9/30): python3-rados-18.2.1-0.el9.x86_64.rpm 13 MB/s | 321 kB 00:00 2024-05-04T12:05:17.644 INFO:teuthology.orchestra.run.smithi079.stdout:(10/30): python3-rbd-18.2.1-0.el9.x86_64.rpm 12 MB/s | 297 kB 00:00 2024-05-04T12:05:17.669 INFO:teuthology.orchestra.run.smithi079.stdout:(11/30): python3-rgw-18.2.1-0.el9.x86_64.rpm 3.9 MB/s | 99 kB 00:00 2024-05-04T12:05:17.745 INFO:teuthology.orchestra.run.smithi079.stdout:(12/30): librgw2-18.2.1-0.el9.x86_64.rpm 14 MB/s | 4.5 MB 00:00 2024-05-04T12:05:18.004 INFO:teuthology.orchestra.run.smithi079.stdout:(13/30): ceph-common-18.2.1-0.el9.x86_64.rpm 21 MB/s | 18 MB 00:00 2024-05-04T12:05:18.126 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: libcephsqlite-2:18.2.1-0.el9.x86_64 3/4 2024-05-04T12:05:18.196 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-test-2:18.2.1-0.el9.x86_64 4/4 2024-05-04T12:05:18.604 INFO:teuthology.orchestra.run.smithi079.stdout:(14/30): boost-program-options-1.75.0-8.el9.x86 114 kB/s | 107 kB 00:00 2024-05-04T12:05:18.654 INFO:teuthology.orchestra.run.smithi079.stdout:(15/30): librabbitmq-0.11.0-7.el9.x86_64.rpm 70 kB/s | 45 kB 00:00 2024-05-04T12:05:18.688 INFO:teuthology.orchestra.run.smithi079.stdout:(16/30): libpmemobj-1.12.1-1.el9.x86_64.rpm 170 kB/s | 160 kB 00:00 2024-05-04T12:05:19.304 INFO:teuthology.orchestra.run.smithi079.stdout:(17/30): python3-prettytable-0.7.2-27.el9.noarc 69 kB/s | 42 kB 00:00 2024-05-04T12:05:19.471 INFO:teuthology.orchestra.run.smithi079.stdout:(18/30): lttng-ust-2.12.0-6.el9.x86_64.rpm 358 kB/s | 292 kB 00:00 2024-05-04T12:05:19.547 INFO:teuthology.orchestra.run.smithi079.stdout:(19/30): librdkafka-1.6.1-102.el9.x86_64.rpm 703 kB/s | 662 kB 00:00 2024-05-04T12:05:19.922 INFO:teuthology.orchestra.run.smithi079.stdout:(20/30): fmt-8.1.1-5.el9.x86_64.rpm 179 kB/s | 111 kB 00:00 2024-05-04T12:05:20.022 INFO:teuthology.orchestra.run.smithi079.stdout:(21/30): libarrow-doc-9.0.0-11.el9.noarch.rpm 260 kB/s | 26 kB 00:00 2024-05-04T12:05:20.161 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-test-2:18.2.1-0.el9.x86_64 4/4 2024-05-04T12:05:20.161 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-test-2:18.2.1-0.el9.x86_64 1/4 2024-05-04T12:05:20.161 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libcephsqlite-2:18.2.1-0.el9.x86_64 2/4 2024-05-04T12:05:20.161 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 3/4 2024-05-04T12:05:20.197 INFO:teuthology.orchestra.run.smithi079.stdout:(22/30): liboath-2.6.7-2.el9.x86_64.rpm 278 kB/s | 49 kB 00:00 2024-05-04T12:05:20.314 INFO:teuthology.orchestra.run.smithi079.stdout:(23/30): gperftools-libs-2.9.1-2.el9.x86_64.rpm 367 kB/s | 309 kB 00:00 2024-05-04T12:05:20.381 INFO:teuthology.orchestra.run.smithi079.stdout:(24/30): libunwind-1.6.2-1.el9.x86_64.rpm 367 kB/s | 67 kB 00:00 2024-05-04T12:05:20.462 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-05-04T12:05:20.462 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-05-04T12:05:20.462 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2024-05-04T12:05:20.462 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-test-2:18.2.1-0.el9.x86_64 libcephsqlite-2:18.2.1-0.el9.x86_64 2024-05-04T12:05:20.462 INFO:teuthology.orchestra.run.smithi146.stdout: socat-1.7.4.1-5.el9.x86_64 xmlstarlet-1.6.1-20.el9.x86_64 2024-05-04T12:05:20.462 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-05-04T12:05:20.462 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-05-04T12:05:20.723 INFO:teuthology.orchestra.run.smithi079.stdout:(25/30): re2-20211101-3.el9.x86_64.rpm 561 kB/s | 192 kB 00:00 2024-05-04T12:05:20.756 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install ceph 2024-05-04T12:05:21.273 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:00:35 ago on Sat 04 May 2024 12:04:46 PM UTC. 2024-05-04T12:05:21.397 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-05-04T12:05:21.398 INFO:teuthology.orchestra.run.smithi079.stdout:(26/30): parquet-libs-9.0.0-11.el9.x86_64.rpm 774 kB/s | 839 kB 00:01 2024-05-04T12:05:21.400 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-05-04T12:05:21.400 INFO:teuthology.orchestra.run.smithi146.stdout: Package Arch Version Repository Size 2024-05-04T12:05:21.400 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-05-04T12:05:21.400 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2024-05-04T12:05:21.400 INFO:teuthology.orchestra.run.smithi146.stdout: ceph x86_64 2:18.2.1-0.el9 ceph 6.4 k 2024-05-04T12:05:21.400 INFO:teuthology.orchestra.run.smithi146.stdout:Installing dependencies: 2024-05-04T12:05:21.400 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mds x86_64 2:18.2.1-0.el9 ceph 2.1 M 2024-05-04T12:05:21.400 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr x86_64 2:18.2.1-0.el9 ceph 1.4 M 2024-05-04T12:05:21.400 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-modules-core noarch 2:18.2.1-0.el9 ceph-noarch 242 k 2024-05-04T12:05:21.400 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mon x86_64 2:18.2.1-0.el9 ceph 4.4 M 2024-05-04T12:05:21.401 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-osd x86_64 2:18.2.1-0.el9 ceph 18 M 2024-05-04T12:05:21.401 INFO:teuthology.orchestra.run.smithi146.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-05-04T12:05:21.401 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-05-04T12:05:21.401 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-05-04T12:05:21.401 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-05-04T12:05:21.401 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-05-04T12:05:21.401 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-05-04T12:05:21.401 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-05-04T12:05:21.401 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-05-04T12:05:21.401 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-05-04T12:05:21.401 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-05-04T12:05:21.402 INFO:teuthology.orchestra.run.smithi146.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-05-04T12:05:21.402 INFO:teuthology.orchestra.run.smithi146.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-05-04T12:05:21.402 INFO:teuthology.orchestra.run.smithi146.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-05-04T12:05:21.402 INFO:teuthology.orchestra.run.smithi146.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-05-04T12:05:21.402 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-05-04T12:05:21.403 INFO:teuthology.orchestra.run.smithi146.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-05-04T12:05:21.403 INFO:teuthology.orchestra.run.smithi146.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-05-04T12:05:21.403 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-05-04T12:05:21.403 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-05-04T12:05:21.403 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-05-04T12:05:21.403 INFO:teuthology.orchestra.run.smithi146.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-05-04T12:05:21.403 INFO:teuthology.orchestra.run.smithi146.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-05-04T12:05:21.403 INFO:teuthology.orchestra.run.smithi146.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-05-04T12:05:21.403 INFO:teuthology.orchestra.run.smithi146.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-05-04T12:05:21.403 INFO:teuthology.orchestra.run.smithi146.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-05-04T12:05:21.403 INFO:teuthology.orchestra.run.smithi146.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-05-04T12:05:21.403 INFO:teuthology.orchestra.run.smithi146.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-05-04T12:05:21.403 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-05-04T12:05:21.404 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-05-04T12:05:21.404 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-05-04T12:05:21.404 INFO:teuthology.orchestra.run.smithi146.stdout:Install 33 Packages 2024-05-04T12:05:21.404 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-05-04T12:05:21.404 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 30 M 2024-05-04T12:05:21.404 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 110 M 2024-05-04T12:05:21.404 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-05-04T12:05:21.474 INFO:teuthology.orchestra.run.smithi079.stdout:(27/30): librados2-18.2.1-0.el9.x86_64.rpm 44 MB/s | 3.3 MB 00:00 2024-05-04T12:05:21.541 INFO:teuthology.orchestra.run.smithi079.stdout:(28/30): librbd1-18.2.1-0.el9.x86_64.rpm 45 MB/s | 3.0 MB 00:00 2024-05-04T12:05:21.941 INFO:teuthology.orchestra.run.smithi146.stdout:(1/33): ceph-18.2.1-0.el9.x86_64.rpm 53 kB/s | 6.4 kB 00:00 2024-05-04T12:05:22.001 INFO:teuthology.orchestra.run.smithi146.stdout:(2/33): ceph-mgr-18.2.1-0.el9.x86_64.rpm 8.1 MB/s | 1.4 MB 00:00 2024-05-04T12:05:22.050 INFO:teuthology.orchestra.run.smithi146.stdout:(3/33): ceph-mds-18.2.1-0.el9.x86_64.rpm 9.2 MB/s | 2.1 MB 00:00 2024-05-04T12:05:22.100 INFO:teuthology.orchestra.run.smithi146.stdout:(4/33): ceph-mgr-modules-core-18.2.1-0.el9.noar 4.7 MB/s | 242 kB 00:00 2024-05-04T12:05:22.167 INFO:teuthology.orchestra.run.smithi146.stdout:(5/33): ceph-mon-18.2.1-0.el9.x86_64.rpm 20 MB/s | 4.4 MB 00:00 2024-05-04T12:05:22.484 INFO:teuthology.orchestra.run.smithi146.stdout:(6/33): ceph-osd-18.2.1-0.el9.x86_64.rpm 36 MB/s | 18 MB 00:00 2024-05-04T12:05:22.509 INFO:teuthology.orchestra.run.smithi146.stdout:(7/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 620 kB/s | 253 kB 00:00 2024-05-04T12:05:22.542 INFO:teuthology.orchestra.run.smithi146.stdout:(8/33): python3-cryptography-36.0.1-4.el9.x86_6 3.3 MB/s | 1.2 MB 00:00 2024-05-04T12:05:22.568 INFO:teuthology.orchestra.run.smithi146.stdout:(9/33): python3-pycparser-2.20-6.el9.noarch.rpm 2.3 MB/s | 135 kB 00:00 2024-05-04T12:05:22.593 INFO:teuthology.orchestra.run.smithi146.stdout:(10/33): python3-requests-2.25.1-8.el9.noarch.r 2.4 MB/s | 125 kB 00:00 2024-05-04T12:05:22.618 INFO:teuthology.orchestra.run.smithi146.stdout:(11/33): python3-urllib3-1.26.5-5.el9.noarch.rp 4.2 MB/s | 215 kB 00:00 2024-05-04T12:05:22.652 INFO:teuthology.orchestra.run.smithi146.stdout:(12/33): python3-ply-3.11-14.el9.noarch.rpm 634 kB/s | 106 kB 00:00 2024-05-04T12:05:22.691 INFO:teuthology.orchestra.run.smithi079.stdout:(29/30): thrift-0.15.0-2.el9.x86_64.rpm 823 kB/s | 1.6 MB 00:01 2024-05-04T12:05:22.811 INFO:teuthology.orchestra.run.smithi146.stdout:(13/33): python3-markupsafe-1.1.1-12.el9.x86_64 181 kB/s | 35 kB 00:00 2024-05-04T12:05:22.844 INFO:teuthology.orchestra.run.smithi146.stdout:(14/33): python3-pytz-2021.1-5.el9.noarch.rpm 266 kB/s | 51 kB 00:00 2024-05-04T12:05:22.886 INFO:teuthology.orchestra.run.smithi146.stdout:(15/33): python3-mako-1.1.4-6.el9.noarch.rpm 589 kB/s | 172 kB 00:00 2024-05-04T12:05:22.911 INFO:teuthology.orchestra.run.smithi146.stdout:(16/33): python3-toml-0.10.2-6.el9.noarch.rpm 415 kB/s | 42 kB 00:00 2024-05-04T12:05:23.470 INFO:teuthology.orchestra.run.smithi146.stdout:(17/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 294 kB/s | 172 kB 00:00 2024-05-04T12:05:23.495 INFO:teuthology.orchestra.run.smithi146.stdout:(18/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 67 kB/s | 43 kB 00:00 2024-05-04T12:05:23.570 INFO:teuthology.orchestra.run.smithi146.stdout:(19/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 107 kB/s | 11 kB 00:00 2024-05-04T12:05:23.679 INFO:teuthology.orchestra.run.smithi146.stdout:(20/33): python3-jaraco-collections-3.0.0-8.el9 214 kB/s | 23 kB 00:00 2024-05-04T12:05:23.704 INFO:teuthology.orchestra.run.smithi146.stdout:(21/33): python3-jaraco-classes-3.2.1-5.el9.noa 84 kB/s | 18 kB 00:00 2024-05-04T12:05:23.821 INFO:teuthology.orchestra.run.smithi146.stdout:(22/33): python3-jaraco-text-3.2.0-6.el9.noarch 168 kB/s | 20 kB 00:00 2024-05-04T12:05:23.871 INFO:teuthology.orchestra.run.smithi146.stdout:(23/33): python3-jaraco-functools-3.5.0-2.el9.n 101 kB/s | 19 kB 00:00 2024-05-04T12:05:23.938 INFO:teuthology.orchestra.run.smithi146.stdout:(24/33): python3-logutils-0.3.5-21.el9.noarch.r 395 kB/s | 46 kB 00:00 2024-05-04T12:05:24.063 INFO:teuthology.orchestra.run.smithi146.stdout:(25/33): python3-cherrypy-18.6.1-2.el9.noarch.r 311 kB/s | 358 kB 00:01 2024-05-04T12:05:24.131 INFO:teuthology.orchestra.run.smithi146.stdout:(26/33): python3-pecan-1.4.2-3.el9.noarch.rpm 1.4 MB/s | 272 kB 00:00 2024-05-04T12:05:24.189 INFO:teuthology.orchestra.run.smithi146.stdout:(27/33): python3-more-itertools-8.12.0-2.el9.no 248 kB/s | 79 kB 00:00 2024-05-04T12:05:24.231 INFO:teuthology.orchestra.run.smithi146.stdout:(28/33): python3-portend-3.1.0-2.el9.noarch.rpm 98 kB/s | 16 kB 00:00 2024-05-04T12:05:24.256 INFO:teuthology.orchestra.run.smithi146.stdout:(29/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 716 kB/s | 90 kB 00:00 2024-05-04T12:05:24.373 INFO:teuthology.orchestra.run.smithi146.stdout:(30/33): python3-tempora-5.0.0-2.el9.noarch.rpm 194 kB/s | 36 kB 00:00 2024-05-04T12:05:24.407 INFO:teuthology.orchestra.run.smithi146.stdout:(31/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.8 MB/s | 427 kB 00:00 2024-05-04T12:05:24.466 INFO:teuthology.orchestra.run.smithi146.stdout:(32/33): python3-webob-1.8.7-6.el9.noarch.rpm 982 kB/s | 230 kB 00:00 2024-05-04T12:05:24.524 INFO:teuthology.orchestra.run.smithi146.stdout:(33/33): python3-zc-lockfile-2.0-10.el9.noarch. 133 kB/s | 20 kB 00:00 2024-05-04T12:05:24.531 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-05-04T12:05:24.531 INFO:teuthology.orchestra.run.smithi146.stdout:Total 9.6 MB/s | 30 MB 00:03 2024-05-04T12:05:24.651 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-05-04T12:05:24.689 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-05-04T12:05:24.690 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-05-04T12:05:24.977 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-05-04T12:05:24.977 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-05-04T12:05:25.240 INFO:teuthology.orchestra.run.smithi079.stdout:(30/30): libarrow-9.0.0-11.el9.x86_64.rpm 794 kB/s | 4.4 MB 00:05 2024-05-04T12:05:25.246 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-05-04T12:05:25.246 INFO:teuthology.orchestra.run.smithi079.stdout:Total 6.3 MB/s | 53 MB 00:08 2024-05-04T12:05:25.320 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-05-04T12:05:25.356 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-05-04T12:05:25.357 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-05-04T12:05:25.778 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-05-04T12:05:25.778 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-05-04T12:05:26.185 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-05-04T12:05:26.252 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-05-04T12:05:26.285 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-05-04T12:05:26.317 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-05-04T12:05:26.410 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-05-04T12:05:26.479 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-05-04T12:05:26.520 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2024-05-04T12:05:26.563 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-05-04T12:05:26.592 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/33 2024-05-04T12:05:26.627 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/33 2024-05-04T12:05:26.660 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/33 2024-05-04T12:05:26.701 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/33 2024-05-04T12:05:26.736 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/33 2024-05-04T12:05:26.770 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/33 2024-05-04T12:05:26.809 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/33 2024-05-04T12:05:26.883 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 15/33 2024-05-04T12:05:27.021 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 16/33 2024-05-04T12:05:27.110 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 17/33 2024-05-04T12:05:27.161 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 18/33 2024-05-04T12:05:27.183 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-05-04T12:05:27.200 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 19/33 2024-05-04T12:05:27.258 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/32 2024-05-04T12:05:27.265 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-ply-3.11-14.el9.noarch 20/33 2024-05-04T12:05:27.295 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/32 2024-05-04T12:05:27.321 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 21/33 2024-05-04T12:05:27.353 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/32 2024-05-04T12:05:27.509 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 22/33 2024-05-04T12:05:27.554 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 23/33 2024-05-04T12:05:27.625 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 24/33 2024-05-04T12:05:27.712 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 25/33 2024-05-04T12:05:27.733 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/32 2024-05-04T12:05:27.744 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 26/33 2024-05-04T12:05:27.786 INFO:teuthology.orchestra.run.smithi079.stdout: Upgrading : librados2-2:18.2.1-0.el9.x86_64 5/32 2024-05-04T12:05:27.860 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 27/33 2024-05-04T12:05:27.878 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: librados2-2:18.2.1-0.el9.x86_64 5/32 2024-05-04T12:05:27.958 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-rados-2:18.2.1-0.el9.x86_64 6/32 2024-05-04T12:05:27.987 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libcephfs2-2:18.2.1-0.el9.x86_64 7/32 2024-05-04T12:05:28.013 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-mgr-modules-core-2:18.2.1-0.el9.noarch 28/33 2024-05-04T12:05:28.050 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: libcephfs2-2:18.2.1-0.el9.x86_64 7/32 2024-05-04T12:05:28.054 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-mgr-2:18.2.1-0.el9.x86_64 29/33 2024-05-04T12:05:28.079 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/32 2024-05-04T12:05:28.079 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-mgr-2:18.2.1-0.el9.x86_64 29/33 2024-05-04T12:05:28.079 INFO:teuthology.orchestra.run.smithi146.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-04T12:05:28.079 INFO:teuthology.orchestra.run.smithi146.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-05-04T12:05:28.079 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-04T12:05:28.079 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-04T12:05:28.080 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-05-04T12:05:28.122 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-ceph-argparse-2:18.2.1-0.el9.x86_64 9/32 2024-05-04T12:05:28.188 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-cephfs-2:18.2.1-0.el9.x86_64 10/32 2024-05-04T12:05:28.214 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libradosstriper1-2:18.2.1-0.el9.x86_64 11/32 2024-05-04T12:05:28.297 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: libradosstriper1-2:18.2.1-0.el9.x86_64 11/32 2024-05-04T12:05:28.403 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/32 2024-05-04T12:05:28.438 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 13/32 2024-05-04T12:05:28.478 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/32 2024-05-04T12:05:28.503 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/32 2024-05-04T12:05:28.530 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 16/32 2024-05-04T12:05:28.566 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : fmt-8.1.1-5.el9.x86_64 17/32 2024-05-04T12:05:28.599 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 18/32 2024-05-04T12:05:28.889 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 19/32 2024-05-04T12:05:28.929 INFO:teuthology.orchestra.run.smithi079.stdout: Upgrading : librbd1-2:18.2.1-0.el9.x86_64 20/32 2024-05-04T12:05:29.019 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: librbd1-2:18.2.1-0.el9.x86_64 20/32 2024-05-04T12:05:29.050 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-rbd-2:18.2.1-0.el9.x86_64 21/32 2024-05-04T12:05:29.085 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 22/32 2024-05-04T12:05:29.491 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 23/32 2024-05-04T12:05:29.523 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : librgw2-2:18.2.1-0.el9.x86_64 24/32 2024-05-04T12:05:29.589 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-osd-2:18.2.1-0.el9.x86_64 30/33 2024-05-04T12:05:29.593 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: librgw2-2:18.2.1-0.el9.x86_64 24/32 2024-05-04T12:05:29.615 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-osd-2:18.2.1-0.el9.x86_64 30/33 2024-05-04T12:05:29.615 INFO:teuthology.orchestra.run.smithi146.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-04T12:05:29.615 INFO:teuthology.orchestra.run.smithi146.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-05-04T12:05:29.615 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-04T12:05:29.616 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-04T12:05:29.616 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-05-04T12:05:29.662 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-rgw-2:18.2.1-0.el9.x86_64 25/32 2024-05-04T12:05:29.692 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-ceph-common-2:18.2.1-0.el9.x86_64 26/32 2024-05-04T12:05:30.065 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-mon-2:18.2.1-0.el9.x86_64 31/33 2024-05-04T12:05:30.088 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-mon-2:18.2.1-0.el9.x86_64 31/33 2024-05-04T12:05:30.088 INFO:teuthology.orchestra.run.smithi146.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-04T12:05:30.088 INFO:teuthology.orchestra.run.smithi146.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-05-04T12:05:30.088 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-04T12:05:30.088 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-04T12:05:30.088 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-05-04T12:05:30.290 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-mds-2:18.2.1-0.el9.x86_64 32/33 2024-05-04T12:05:30.314 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-mds-2:18.2.1-0.el9.x86_64 32/33 2024-05-04T12:05:30.314 INFO:teuthology.orchestra.run.smithi146.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-04T12:05:30.314 INFO:teuthology.orchestra.run.smithi146.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-05-04T12:05:30.314 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-04T12:05:30.314 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-04T12:05:30.314 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-05-04T12:05:30.377 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-2:18.2.1-0.el9.x86_64 33/33 2024-05-04T12:05:31.658 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-common-2:18.2.1-0.el9.x86_64 27/32 2024-05-04T12:05:31.721 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-common-2:18.2.1-0.el9.x86_64 27/32 2024-05-04T12:05:32.151 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-2:18.2.1-0.el9.x86_64 33/33 2024-05-04T12:05:32.151 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-2:18.2.1-0.el9.x86_64 1/33 2024-05-04T12:05:32.151 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-mds-2:18.2.1-0.el9.x86_64 2/33 2024-05-04T12:05:32.151 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-mgr-2:18.2.1-0.el9.x86_64 3/33 2024-05-04T12:05:32.151 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-mon-2:18.2.1-0.el9.x86_64 4/33 2024-05-04T12:05:32.151 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-osd-2:18.2.1-0.el9.x86_64 5/33 2024-05-04T12:05:32.151 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-mgr-modules-core-2:18.2.1-0.el9.noarch 6/33 2024-05-04T12:05:32.152 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/33 2024-05-04T12:05:32.152 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/33 2024-05-04T12:05:32.152 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/33 2024-05-04T12:05:32.152 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/33 2024-05-04T12:05:32.152 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/33 2024-05-04T12:05:32.152 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 12/33 2024-05-04T12:05:32.152 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/33 2024-05-04T12:05:32.152 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/33 2024-05-04T12:05:32.152 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 15/33 2024-05-04T12:05:32.152 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 16/33 2024-05-04T12:05:32.152 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-05-04T12:05:32.152 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-05-04T12:05:32.152 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-05-04T12:05:32.153 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-05-04T12:05:32.153 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-05-04T12:05:32.153 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-05-04T12:05:32.153 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-05-04T12:05:32.153 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-05-04T12:05:32.153 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-05-04T12:05:32.153 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-05-04T12:05:32.153 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-05-04T12:05:32.153 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-05-04T12:05:32.153 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-05-04T12:05:32.153 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-05-04T12:05:32.153 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2024-05-04T12:05:32.153 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-05-04T12:05:32.258 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-common-2:18.2.1-0.el9.x86_64 27/32 2024-05-04T12:05:32.300 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-base-2:18.2.1-0.el9.x86_64 28/32 2024-05-04T12:05:32.397 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-base-2:18.2.1-0.el9.x86_64 28/32 2024-05-04T12:05:32.397 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-05-04T12:05:32.397 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-05-04T12:05:32.397 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-05-04T12:05:32.426 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-selinux-2:18.2.1-0.el9.x86_64 29/32 2024-05-04T12:05:33.138 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-05-04T12:05:33.138 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-05-04T12:05:33.139 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2024-05-04T12:05:33.139 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-2:18.2.1-0.el9.x86_64 2024-05-04T12:05:33.139 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mds-2:18.2.1-0.el9.x86_64 2024-05-04T12:05:33.139 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-2:18.2.1-0.el9.x86_64 2024-05-04T12:05:33.139 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-modules-core-2:18.2.1-0.el9.noarch 2024-05-04T12:05:33.139 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mon-2:18.2.1-0.el9.x86_64 2024-05-04T12:05:33.139 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-osd-2:18.2.1-0.el9.x86_64 2024-05-04T12:05:33.139 INFO:teuthology.orchestra.run.smithi146.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-05-04T12:05:33.139 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-05-04T12:05:33.139 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-05-04T12:05:33.139 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-05-04T12:05:33.139 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-05-04T12:05:33.139 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-05-04T12:05:33.140 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-05-04T12:05:33.140 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-05-04T12:05:33.140 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-05-04T12:05:33.140 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-05-04T12:05:33.140 INFO:teuthology.orchestra.run.smithi146.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-05-04T12:05:33.140 INFO:teuthology.orchestra.run.smithi146.stdout: python3-mako-1.1.4-6.el9.noarch 2024-05-04T12:05:33.140 INFO:teuthology.orchestra.run.smithi146.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-05-04T12:05:33.140 INFO:teuthology.orchestra.run.smithi146.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-05-04T12:05:33.140 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-05-04T12:05:33.140 INFO:teuthology.orchestra.run.smithi146.stdout: python3-ply-3.11-14.el9.noarch 2024-05-04T12:05:33.140 INFO:teuthology.orchestra.run.smithi146.stdout: python3-portend-3.1.0-2.el9.noarch 2024-05-04T12:05:33.140 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-05-04T12:05:33.140 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pycparser-2.20-6.el9.noarch 2024-05-04T12:05:33.140 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pytz-2021.1-5.el9.noarch 2024-05-04T12:05:33.141 INFO:teuthology.orchestra.run.smithi146.stdout: python3-requests-2.25.1-8.el9.noarch 2024-05-04T12:05:33.141 INFO:teuthology.orchestra.run.smithi146.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-05-04T12:05:33.141 INFO:teuthology.orchestra.run.smithi146.stdout: python3-toml-0.10.2-6.el9.noarch 2024-05-04T12:05:33.141 INFO:teuthology.orchestra.run.smithi146.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-05-04T12:05:33.141 INFO:teuthology.orchestra.run.smithi146.stdout: python3-webob-1.8.7-6.el9.noarch 2024-05-04T12:05:33.141 INFO:teuthology.orchestra.run.smithi146.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-05-04T12:05:33.141 INFO:teuthology.orchestra.run.smithi146.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-05-04T12:05:33.141 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-05-04T12:05:33.141 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-05-04T12:05:33.476 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install ceph-base 2024-05-04T12:05:33.989 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:00:47 ago on Sat 04 May 2024 12:04:46 PM UTC. 2024-05-04T12:05:34.069 INFO:teuthology.orchestra.run.smithi146.stdout:Package ceph-base-2:18.2.1-0.el9.x86_64 is already installed. 2024-05-04T12:05:34.109 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-05-04T12:05:34.110 INFO:teuthology.orchestra.run.smithi146.stdout:Nothing to do. 2024-05-04T12:05:34.111 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-05-04T12:05:34.176 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install cephadm 2024-05-04T12:05:34.684 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:00:48 ago on Sat 04 May 2024 12:04:46 PM UTC. 2024-05-04T12:05:34.804 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-05-04T12:05:34.805 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-05-04T12:05:34.805 INFO:teuthology.orchestra.run.smithi146.stdout: Package Architecture Version Repository Size 2024-05-04T12:05:34.805 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-05-04T12:05:34.805 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2024-05-04T12:05:34.805 INFO:teuthology.orchestra.run.smithi146.stdout: cephadm noarch 2:18.2.1-0.el9 ceph-noarch 221 k 2024-05-04T12:05:34.805 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-05-04T12:05:34.805 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-05-04T12:05:34.805 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-05-04T12:05:34.806 INFO:teuthology.orchestra.run.smithi146.stdout:Install 1 Package 2024-05-04T12:05:34.806 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-05-04T12:05:34.806 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 221 k 2024-05-04T12:05:34.806 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 213 k 2024-05-04T12:05:34.806 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-05-04T12:05:34.906 INFO:teuthology.orchestra.run.smithi146.stdout:cephadm-18.2.1-0.el9.noarch.rpm 2.2 MB/s | 221 kB 00:00 2024-05-04T12:05:34.907 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-05-04T12:05:34.907 INFO:teuthology.orchestra.run.smithi146.stdout:Total 2.1 MB/s | 221 kB 00:00 2024-05-04T12:05:34.907 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-05-04T12:05:34.912 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-05-04T12:05:34.912 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-05-04T12:05:34.914 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-05-04T12:05:34.915 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-05-04T12:05:34.983 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-05-04T12:05:35.399 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: cephadm-2:18.2.1-0.el9.noarch 1/1 2024-05-04T12:05:35.493 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : cephadm-2:18.2.1-0.el9.noarch 1/1 2024-05-04T12:05:35.837 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: cephadm-2:18.2.1-0.el9.noarch 1/1 2024-05-04T12:05:36.077 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : cephadm-2:18.2.1-0.el9.noarch 1/1 2024-05-04T12:05:36.078 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-05-04T12:05:36.078 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2024-05-04T12:05:36.078 INFO:teuthology.orchestra.run.smithi146.stdout: cephadm-2:18.2.1-0.el9.noarch 2024-05-04T12:05:36.078 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-05-04T12:05:36.078 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-05-04T12:05:36.268 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install ceph-immutable-object-cache 2024-05-04T12:05:36.794 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:00:50 ago on Sat 04 May 2024 12:04:46 PM UTC. 2024-05-04T12:05:36.921 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-05-04T12:05:36.921 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-05-04T12:05:36.921 INFO:teuthology.orchestra.run.smithi146.stdout: Package Arch Version Repo Size 2024-05-04T12:05:36.921 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-05-04T12:05:36.922 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2024-05-04T12:05:36.922 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-immutable-object-cache x86_64 2:18.2.1-0.el9 ceph 142 k 2024-05-04T12:05:36.922 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-05-04T12:05:36.922 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-05-04T12:05:36.922 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-05-04T12:05:36.922 INFO:teuthology.orchestra.run.smithi146.stdout:Install 1 Package 2024-05-04T12:05:36.922 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-05-04T12:05:36.923 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 142 k 2024-05-04T12:05:36.923 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 395 k 2024-05-04T12:05:36.923 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-05-04T12:05:37.020 INFO:teuthology.orchestra.run.smithi146.stdout:ceph-immutable-object-cache-18.2.1-0.el9.x86_64 1.4 MB/s | 142 kB 00:00 2024-05-04T12:05:37.021 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-05-04T12:05:37.021 INFO:teuthology.orchestra.run.smithi146.stdout:Total 1.4 MB/s | 142 kB 00:00 2024-05-04T12:05:37.022 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-05-04T12:05:37.030 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-05-04T12:05:37.030 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-05-04T12:05:37.089 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-05-04T12:05:37.089 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-05-04T12:05:37.240 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-05-04T12:05:37.314 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-immutable-object-cache-2:18.2.1-0.el9.x86_64 1/1 2024-05-04T12:05:37.339 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.1-0.el9.x86_64 1/1 2024-05-04T12:05:37.339 INFO:teuthology.orchestra.run.smithi146.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-04T12:05:37.339 INFO:teuthology.orchestra.run.smithi146.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-05-04T12:05:37.340 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-05-04T12:05:38.233 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-immutable-object-cache-2:18.2.1-0.el9.x86_64 1/1 2024-05-04T12:05:38.234 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-05-04T12:05:38.234 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2024-05-04T12:05:38.234 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-immutable-object-cache-2:18.2.1-0.el9.x86_64 2024-05-04T12:05:38.234 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-05-04T12:05:38.234 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-05-04T12:05:38.460 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install ceph-mgr 2024-05-04T12:05:38.978 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:00:52 ago on Sat 04 May 2024 12:04:46 PM UTC. 2024-05-04T12:05:39.058 INFO:teuthology.orchestra.run.smithi146.stdout:Package ceph-mgr-2:18.2.1-0.el9.x86_64 is already installed. 2024-05-04T12:05:39.100 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-05-04T12:05:39.101 INFO:teuthology.orchestra.run.smithi146.stdout:Nothing to do. 2024-05-04T12:05:39.101 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-05-04T12:05:39.159 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install ceph-mgr-dashboard 2024-05-04T12:05:39.671 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:00:53 ago on Sat 04 May 2024 12:04:46 PM UTC. 2024-05-04T12:05:39.790 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-05-04T12:05:39.791 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-05-04T12:05:39.791 INFO:teuthology.orchestra.run.smithi146.stdout: Package Arch Version Repository Size 2024-05-04T12:05:39.791 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-05-04T12:05:39.791 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2024-05-04T12:05:39.791 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-dashboard noarch 2:18.2.1-0.el9 ceph-noarch 1.8 M 2024-05-04T12:05:39.791 INFO:teuthology.orchestra.run.smithi146.stdout:Installing dependencies: 2024-05-04T12:05:39.791 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-grafana-dashboards noarch 2:18.2.1-0.el9 ceph-noarch 23 k 2024-05-04T12:05:39.792 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-prometheus-alerts noarch 2:18.2.1-0.el9 ceph-noarch 15 k 2024-05-04T12:05:39.792 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-05-04T12:05:39.792 INFO:teuthology.orchestra.run.smithi146.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-05-04T12:05:39.792 INFO:teuthology.orchestra.run.smithi146.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-05-04T12:05:39.792 INFO:teuthology.orchestra.run.smithi146.stdout:Installing weak dependencies: 2024-05-04T12:05:39.792 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-05-04T12:05:39.792 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-05-04T12:05:39.792 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-05-04T12:05:39.792 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-05-04T12:05:39.792 INFO:teuthology.orchestra.run.smithi146.stdout:Install 7 Packages 2024-05-04T12:05:39.792 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-05-04T12:05:39.793 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 2.1 M 2024-05-04T12:05:39.793 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 8.5 M 2024-05-04T12:05:39.793 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-05-04T12:05:40.068 INFO:teuthology.orchestra.run.smithi146.stdout:(1/7): ceph-grafana-dashboards-18.2.1-0.el9.noa 181 kB/s | 23 kB 00:00 2024-05-04T12:05:40.177 INFO:teuthology.orchestra.run.smithi146.stdout:(2/7): ceph-mgr-dashboard-18.2.1-0.el9.noarch.r 7.4 MB/s | 1.8 MB 00:00 2024-05-04T12:05:40.202 INFO:teuthology.orchestra.run.smithi146.stdout:(3/7): ceph-prometheus-alerts-18.2.1-0.el9.noar 56 kB/s | 15 kB 00:00 2024-05-04T12:05:40.227 INFO:teuthology.orchestra.run.smithi146.stdout:(4/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 57 kB/s | 9.0 kB 00:00 2024-05-04T12:05:40.294 INFO:teuthology.orchestra.run.smithi146.stdout:(5/7): python3-jwt-2.4.0-1.el9.noarch.rpm 350 kB/s | 41 kB 00:00 2024-05-04T12:05:40.378 INFO:teuthology.orchestra.run.smithi146.stdout:(6/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 205 kB/s | 31 kB 00:00 2024-05-04T12:05:40.495 INFO:teuthology.orchestra.run.smithi146.stdout:(7/7): python3-routes-2.5.1-5.el9.noarch.rpm 706 kB/s | 188 kB 00:00 2024-05-04T12:05:40.496 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-05-04T12:05:40.497 INFO:teuthology.orchestra.run.smithi146.stdout:Total 2.9 MB/s | 2.1 MB 00:00 2024-05-04T12:05:40.497 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-05-04T12:05:40.505 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-05-04T12:05:40.505 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-05-04T12:05:40.555 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-05-04T12:05:40.555 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-05-04T12:05:40.837 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-05-04T12:05:40.896 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-05-04T12:05:40.930 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-05-04T12:05:40.984 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-05-04T12:05:41.025 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-05-04T12:05:41.054 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-prometheus-alerts-2:18.2.1-0.el9.noarch 5/7 2024-05-04T12:05:41.291 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-grafana-dashboards-2:18.2.1-0.el9.noarch 6/7 2024-05-04T12:05:41.324 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-mgr-dashboard-2:18.2.1-0.el9.noarch 7/7 2024-05-04T12:05:41.756 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-selinux-2:18.2.1-0.el9.x86_64 29/32 2024-05-04T12:05:41.757 INFO:teuthology.orchestra.run.smithi079.stdout:skipping the directory /sys 2024-05-04T12:05:41.757 INFO:teuthology.orchestra.run.smithi079.stdout:skipping the directory /proc 2024-05-04T12:05:41.757 INFO:teuthology.orchestra.run.smithi079.stdout:skipping the directory /mnt 2024-05-04T12:05:41.757 INFO:teuthology.orchestra.run.smithi079.stdout:skipping the directory /var/tmp 2024-05-04T12:05:41.757 INFO:teuthology.orchestra.run.smithi079.stdout:skipping the directory /home 2024-05-04T12:05:41.757 INFO:teuthology.orchestra.run.smithi079.stdout:skipping the directory /root 2024-05-04T12:05:41.757 INFO:teuthology.orchestra.run.smithi079.stdout:skipping the directory /tmp 2024-05-04T12:05:41.757 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-05-04T12:05:41.861 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.1-0.el9.noarch 7/7 2024-05-04T12:05:41.861 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-grafana-dashboards-2:18.2.1-0.el9.noarch 1/7 2024-05-04T12:05:41.861 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-mgr-dashboard-2:18.2.1-0.el9.noarch 2/7 2024-05-04T12:05:41.861 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-prometheus-alerts-2:18.2.1-0.el9.noarch 3/7 2024-05-04T12:05:41.861 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-05-04T12:05:41.861 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-05-04T12:05:41.861 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-05-04T12:05:42.117 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-05-04T12:05:42.118 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-05-04T12:05:42.118 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2024-05-04T12:05:42.118 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-grafana-dashboards-2:18.2.1-0.el9.noarch 2024-05-04T12:05:42.118 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-dashboard-2:18.2.1-0.el9.noarch 2024-05-04T12:05:42.118 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-prometheus-alerts-2:18.2.1-0.el9.noarch 2024-05-04T12:05:42.118 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-05-04T12:05:42.118 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-05-04T12:05:42.118 INFO:teuthology.orchestra.run.smithi146.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-05-04T12:05:42.118 INFO:teuthology.orchestra.run.smithi146.stdout: python3-routes-2.5.1-5.el9.noarch 2024-05-04T12:05:42.119 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-05-04T12:05:42.119 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-05-04T12:05:42.329 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install ceph-mgr-diskprediction-local 2024-05-04T12:05:42.475 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-radosgw-2:18.2.1-0.el9.x86_64 30/32 2024-05-04T12:05:42.500 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-radosgw-2:18.2.1-0.el9.x86_64 30/32 2024-05-04T12:05:42.500 INFO:teuthology.orchestra.run.smithi079.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-04T12:05:42.500 INFO:teuthology.orchestra.run.smithi079.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-05-04T12:05:42.500 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-04T12:05:42.500 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-04T12:05:42.500 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-05-04T12:05:42.502 INFO:teuthology.orchestra.run.smithi079.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-05-04T12:05:42.569 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-05-04T12:05:42.570 INFO:teuthology.orchestra.run.smithi079.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 32/32 2024-05-04T12:05:42.840 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:00:56 ago on Sat 04 May 2024 12:04:46 PM UTC. 2024-05-04T12:05:42.959 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-05-04T12:05:42.960 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-05-04T12:05:42.960 INFO:teuthology.orchestra.run.smithi146.stdout: Package Arch Version Repository Size 2024-05-04T12:05:42.960 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-05-04T12:05:42.960 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2024-05-04T12:05:42.960 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.1-0.el9 ceph-noarch 7.4 M 2024-05-04T12:05:42.960 INFO:teuthology.orchestra.run.smithi146.stdout:Installing dependencies: 2024-05-04T12:05:42.960 INFO:teuthology.orchestra.run.smithi146.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-05-04T12:05:42.960 INFO:teuthology.orchestra.run.smithi146.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-05-04T12:05:42.960 INFO:teuthology.orchestra.run.smithi146.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-05-04T12:05:42.960 INFO:teuthology.orchestra.run.smithi146.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-05-04T12:05:42.961 INFO:teuthology.orchestra.run.smithi146.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-05-04T12:05:42.961 INFO:teuthology.orchestra.run.smithi146.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-05-04T12:05:42.961 INFO:teuthology.orchestra.run.smithi146.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-05-04T12:05:42.961 INFO:teuthology.orchestra.run.smithi146.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-05-04T12:05:42.961 INFO:teuthology.orchestra.run.smithi146.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-05-04T12:05:42.961 INFO:teuthology.orchestra.run.smithi146.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-05-04T12:05:42.961 INFO:teuthology.orchestra.run.smithi146.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-05-04T12:05:42.961 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-05-04T12:05:42.961 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-05-04T12:05:42.961 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-05-04T12:05:42.961 INFO:teuthology.orchestra.run.smithi146.stdout:Install 12 Packages 2024-05-04T12:05:42.961 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-05-04T12:05:42.962 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 38 M 2024-05-04T12:05:42.962 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 209 M 2024-05-04T12:05:42.962 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-05-04T12:05:43.635 INFO:teuthology.orchestra.run.smithi146.stdout:(1/12): ceph-mgr-diskprediction-local-18.2.1-0. 17 MB/s | 7.4 MB 00:00 2024-05-04T12:05:43.702 INFO:teuthology.orchestra.run.smithi146.stdout:(2/12): libquadmath-11.4.1-3.el9.x86_64.rpm 389 kB/s | 195 kB 00:00 2024-05-04T12:05:43.742 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 32/32 2024-05-04T12:05:43.742 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-base-2:18.2.1-0.el9.x86_64 1/32 2024-05-04T12:05:43.742 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-common-2:18.2.1-0.el9.x86_64 2/32 2024-05-04T12:05:43.742 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-radosgw-2:18.2.1-0.el9.x86_64 3/32 2024-05-04T12:05:43.742 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-selinux-2:18.2.1-0.el9.x86_64 4/32 2024-05-04T12:05:43.742 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libcephfs2-2:18.2.1-0.el9.x86_64 5/32 2024-05-04T12:05:43.743 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libradosstriper1-2:18.2.1-0.el9.x86_64 6/32 2024-05-04T12:05:43.743 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : librgw2-2:18.2.1-0.el9.x86_64 7/32 2024-05-04T12:05:43.743 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-ceph-argparse-2:18.2.1-0.el9.x86_64 8/32 2024-05-04T12:05:43.743 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-ceph-common-2:18.2.1-0.el9.x86_64 9/32 2024-05-04T12:05:43.743 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-cephfs-2:18.2.1-0.el9.x86_64 10/32 2024-05-04T12:05:43.743 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-rados-2:18.2.1-0.el9.x86_64 11/32 2024-05-04T12:05:43.743 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-rbd-2:18.2.1-0.el9.x86_64 12/32 2024-05-04T12:05:43.743 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-rgw-2:18.2.1-0.el9.x86_64 13/32 2024-05-04T12:05:43.743 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/32 2024-05-04T12:05:43.743 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/32 2024-05-04T12:05:43.743 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/32 2024-05-04T12:05:43.743 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/32 2024-05-04T12:05:43.743 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/32 2024-05-04T12:05:43.743 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/32 2024-05-04T12:05:43.744 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 20/32 2024-05-04T12:05:43.744 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 21/32 2024-05-04T12:05:43.744 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 22/32 2024-05-04T12:05:43.744 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 23/32 2024-05-04T12:05:43.744 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 24/32 2024-05-04T12:05:43.744 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 25/32 2024-05-04T12:05:43.744 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 26/32 2024-05-04T12:05:43.744 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : re2-1:20211101-3.el9.x86_64 27/32 2024-05-04T12:05:43.744 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 28/32 2024-05-04T12:05:43.744 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : librados2-2:18.2.1-0.el9.x86_64 29/32 2024-05-04T12:05:43.744 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 30/32 2024-05-04T12:05:43.744 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : librbd1-2:18.2.1-0.el9.x86_64 31/32 2024-05-04T12:05:43.902 INFO:teuthology.orchestra.run.smithi146.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 122 kB/s | 33 kB 00:00 2024-05-04T12:05:44.002 INFO:teuthology.orchestra.run.smithi146.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 182 kB/s | 18 kB 00:00 2024-05-04T12:05:44.027 INFO:teuthology.orchestra.run.smithi146.stdout:(5/12): libgfortran-11.4.1-3.el9.x86_64.rpm 970 kB/s | 803 kB 00:00 2024-05-04T12:05:44.106 INFO:teuthology.orchestra.run.smithi146.stdout:(6/12): openblas-0.3.21-2.el9.x86_64.rpm 350 kB/s | 35 kB 00:00 2024-05-04T12:05:44.178 INFO:teuthology.orchestra.run.smithi146.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 6.3 MB/s | 3.0 MB 00:00 2024-05-04T12:05:44.378 INFO:teuthology.orchestra.run.smithi146.stdout:(8/12): python3-devel-3.9.18-3.el9.x86_64.rpm 887 kB/s | 244 kB 00:00 2024-05-04T12:05:44.485 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 32/32 2024-05-04T12:05:44.485 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-05-04T12:05:44.485 INFO:teuthology.orchestra.run.smithi079.stdout:Upgraded: 2024-05-04T12:05:44.485 INFO:teuthology.orchestra.run.smithi079.stdout: librados2-2:18.2.1-0.el9.x86_64 librbd1-2:18.2.1-0.el9.x86_64 2024-05-04T12:05:44.485 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-05-04T12:05:44.485 INFO:teuthology.orchestra.run.smithi079.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-05-04T12:05:44.485 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-base-2:18.2.1-0.el9.x86_64 2024-05-04T12:05:44.485 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-common-2:18.2.1-0.el9.x86_64 2024-05-04T12:05:44.485 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-radosgw-2:18.2.1-0.el9.x86_64 2024-05-04T12:05:44.485 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-selinux-2:18.2.1-0.el9.x86_64 2024-05-04T12:05:44.485 INFO:teuthology.orchestra.run.smithi079.stdout: fmt-8.1.1-5.el9.x86_64 2024-05-04T12:05:44.485 INFO:teuthology.orchestra.run.smithi079.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-05-04T12:05:44.485 INFO:teuthology.orchestra.run.smithi079.stdout: libarrow-9.0.0-11.el9.x86_64 2024-05-04T12:05:44.486 INFO:teuthology.orchestra.run.smithi079.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-05-04T12:05:44.486 INFO:teuthology.orchestra.run.smithi079.stdout: libcephfs2-2:18.2.1-0.el9.x86_64 2024-05-04T12:05:44.486 INFO:teuthology.orchestra.run.smithi079.stdout: liboath-2.6.7-2.el9.x86_64 2024-05-04T12:05:44.486 INFO:teuthology.orchestra.run.smithi079.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-05-04T12:05:44.486 INFO:teuthology.orchestra.run.smithi079.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-05-04T12:05:44.486 INFO:teuthology.orchestra.run.smithi079.stdout: libradosstriper1-2:18.2.1-0.el9.x86_64 2024-05-04T12:05:44.486 INFO:teuthology.orchestra.run.smithi079.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-05-04T12:05:44.486 INFO:teuthology.orchestra.run.smithi079.stdout: librgw2-2:18.2.1-0.el9.x86_64 2024-05-04T12:05:44.486 INFO:teuthology.orchestra.run.smithi079.stdout: libunwind-1.6.2-1.el9.x86_64 2024-05-04T12:05:44.486 INFO:teuthology.orchestra.run.smithi079.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-05-04T12:05:44.486 INFO:teuthology.orchestra.run.smithi079.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-05-04T12:05:44.486 INFO:teuthology.orchestra.run.smithi079.stdout: python3-ceph-argparse-2:18.2.1-0.el9.x86_64 2024-05-04T12:05:44.486 INFO:teuthology.orchestra.run.smithi079.stdout: python3-ceph-common-2:18.2.1-0.el9.x86_64 2024-05-04T12:05:44.487 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cephfs-2:18.2.1-0.el9.x86_64 2024-05-04T12:05:44.487 INFO:teuthology.orchestra.run.smithi079.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-05-04T12:05:44.487 INFO:teuthology.orchestra.run.smithi079.stdout: python3-rados-2:18.2.1-0.el9.x86_64 2024-05-04T12:05:44.487 INFO:teuthology.orchestra.run.smithi079.stdout: python3-rbd-2:18.2.1-0.el9.x86_64 2024-05-04T12:05:44.487 INFO:teuthology.orchestra.run.smithi079.stdout: python3-rgw-2:18.2.1-0.el9.x86_64 2024-05-04T12:05:44.487 INFO:teuthology.orchestra.run.smithi079.stdout: re2-1:20211101-3.el9.x86_64 2024-05-04T12:05:44.487 INFO:teuthology.orchestra.run.smithi079.stdout: thrift-0.15.0-2.el9.x86_64 2024-05-04T12:05:44.487 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-05-04T12:05:44.487 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-05-04T12:05:44.579 INFO:teuthology.orchestra.run.smithi146.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 1.2 MB/s | 250 kB 00:00 2024-05-04T12:05:44.723 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install ceph-test 2024-05-04T12:05:44.962 INFO:teuthology.orchestra.run.smithi146.stdout:(10/12): openblas-openmp-0.3.21-2.el9.x86_64.rp 5.1 MB/s | 4.7 MB 00:00 2024-05-04T12:05:45.260 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:00:31 ago on Sat 04 May 2024 12:05:14 PM UTC. 2024-05-04T12:05:45.380 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-05-04T12:05:45.381 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-05-04T12:05:45.381 INFO:teuthology.orchestra.run.smithi079.stdout: Package Architecture Version Repository Size 2024-05-04T12:05:45.381 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-05-04T12:05:45.381 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-05-04T12:05:45.381 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-test x86_64 2:18.2.1-0.el9 ceph 40 M 2024-05-04T12:05:45.381 INFO:teuthology.orchestra.run.smithi079.stdout:Installing dependencies: 2024-05-04T12:05:45.381 INFO:teuthology.orchestra.run.smithi079.stdout: libcephsqlite x86_64 2:18.2.1-0.el9 ceph 165 k 2024-05-04T12:05:45.381 INFO:teuthology.orchestra.run.smithi079.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-05-04T12:05:45.381 INFO:teuthology.orchestra.run.smithi079.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-05-04T12:05:45.382 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-05-04T12:05:45.382 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-05-04T12:05:45.382 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-05-04T12:05:45.382 INFO:teuthology.orchestra.run.smithi079.stdout:Install 4 Packages 2024-05-04T12:05:45.382 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-05-04T12:05:45.382 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 40 M 2024-05-04T12:05:45.383 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 166 M 2024-05-04T12:05:45.383 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-05-04T12:05:45.560 INFO:teuthology.orchestra.run.smithi079.stdout:(1/4): libcephsqlite-18.2.1-0.el9.x86_64.rpm 1.5 MB/s | 165 kB 00:00 2024-05-04T12:05:45.677 INFO:teuthology.orchestra.run.smithi079.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 542 kB/s | 64 kB 00:00 2024-05-04T12:05:45.711 INFO:teuthology.orchestra.run.smithi079.stdout:(3/4): socat-1.7.4.1-5.el9.x86_64.rpm 1.2 MB/s | 305 kB 00:00 2024-05-04T12:05:46.146 INFO:teuthology.orchestra.run.smithi079.stdout:(4/4): ceph-test-18.2.1-0.el9.x86_64.rpm 57 MB/s | 40 MB 00:00 2024-05-04T12:05:46.148 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-05-04T12:05:46.149 INFO:teuthology.orchestra.run.smithi079.stdout:Total 52 MB/s | 40 MB 00:00 2024-05-04T12:05:46.174 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-05-04T12:05:46.194 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-05-04T12:05:46.194 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-05-04T12:05:46.516 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-05-04T12:05:46.517 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-05-04T12:05:47.033 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-05-04T12:05:47.139 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-05-04T12:05:47.204 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/4 2024-05-04T12:05:47.265 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libcephsqlite-2:18.2.1-0.el9.x86_64 3/4 2024-05-04T12:05:48.687 INFO:teuthology.orchestra.run.smithi146.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 1.1 MB/s | 5.1 MB 00:04 2024-05-04T12:05:50.987 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: libcephsqlite-2:18.2.1-0.el9.x86_64 3/4 2024-05-04T12:05:51.098 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-test-2:18.2.1-0.el9.x86_64 4/4 2024-05-04T12:05:53.175 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-test-2:18.2.1-0.el9.x86_64 4/4 2024-05-04T12:05:53.175 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-test-2:18.2.1-0.el9.x86_64 1/4 2024-05-04T12:05:53.175 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libcephsqlite-2:18.2.1-0.el9.x86_64 2/4 2024-05-04T12:05:53.175 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 3/4 2024-05-04T12:05:53.434 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-05-04T12:05:53.435 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-05-04T12:05:53.435 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-05-04T12:05:53.435 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-test-2:18.2.1-0.el9.x86_64 libcephsqlite-2:18.2.1-0.el9.x86_64 2024-05-04T12:05:53.435 INFO:teuthology.orchestra.run.smithi079.stdout: socat-1.7.4.1-5.el9.x86_64 xmlstarlet-1.6.1-20.el9.x86_64 2024-05-04T12:05:53.435 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-05-04T12:05:53.435 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-05-04T12:05:53.706 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install ceph 2024-05-04T12:05:54.215 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:00:40 ago on Sat 04 May 2024 12:05:14 PM UTC. 2024-05-04T12:05:54.343 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-05-04T12:05:54.343 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-05-04T12:05:54.343 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repository Size 2024-05-04T12:05:54.343 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-05-04T12:05:54.343 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-05-04T12:05:54.343 INFO:teuthology.orchestra.run.smithi079.stdout: ceph x86_64 2:18.2.1-0.el9 ceph 6.4 k 2024-05-04T12:05:54.343 INFO:teuthology.orchestra.run.smithi079.stdout:Installing dependencies: 2024-05-04T12:05:54.343 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mds x86_64 2:18.2.1-0.el9 ceph 2.1 M 2024-05-04T12:05:54.344 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr x86_64 2:18.2.1-0.el9 ceph 1.4 M 2024-05-04T12:05:54.344 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-modules-core noarch 2:18.2.1-0.el9 ceph-noarch 242 k 2024-05-04T12:05:54.344 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mon x86_64 2:18.2.1-0.el9 ceph 4.4 M 2024-05-04T12:05:54.344 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-osd x86_64 2:18.2.1-0.el9 ceph 18 M 2024-05-04T12:05:54.344 INFO:teuthology.orchestra.run.smithi079.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-05-04T12:05:54.344 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-05-04T12:05:54.344 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-05-04T12:05:54.344 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-05-04T12:05:54.344 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-05-04T12:05:54.344 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-05-04T12:05:54.344 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-05-04T12:05:54.344 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-05-04T12:05:54.344 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-05-04T12:05:54.345 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-05-04T12:05:54.345 INFO:teuthology.orchestra.run.smithi079.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-05-04T12:05:54.345 INFO:teuthology.orchestra.run.smithi079.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-05-04T12:05:54.345 INFO:teuthology.orchestra.run.smithi079.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-05-04T12:05:54.345 INFO:teuthology.orchestra.run.smithi079.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-05-04T12:05:54.345 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-05-04T12:05:54.345 INFO:teuthology.orchestra.run.smithi079.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-05-04T12:05:54.345 INFO:teuthology.orchestra.run.smithi079.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-05-04T12:05:54.345 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-05-04T12:05:54.345 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-05-04T12:05:54.345 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-05-04T12:05:54.345 INFO:teuthology.orchestra.run.smithi079.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-05-04T12:05:54.346 INFO:teuthology.orchestra.run.smithi079.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-05-04T12:05:54.346 INFO:teuthology.orchestra.run.smithi079.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-05-04T12:05:54.346 INFO:teuthology.orchestra.run.smithi079.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-05-04T12:05:54.346 INFO:teuthology.orchestra.run.smithi079.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-05-04T12:05:54.346 INFO:teuthology.orchestra.run.smithi079.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-05-04T12:05:54.346 INFO:teuthology.orchestra.run.smithi079.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-05-04T12:05:54.346 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-05-04T12:05:54.346 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-05-04T12:05:54.346 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-05-04T12:05:54.346 INFO:teuthology.orchestra.run.smithi079.stdout:Install 33 Packages 2024-05-04T12:05:54.346 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-05-04T12:05:54.347 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 30 M 2024-05-04T12:05:54.347 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 110 M 2024-05-04T12:05:54.347 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-05-04T12:05:54.858 INFO:teuthology.orchestra.run.smithi079.stdout:(1/33): ceph-18.2.1-0.el9.x86_64.rpm 58 kB/s | 6.4 kB 00:00 2024-05-04T12:05:54.908 INFO:teuthology.orchestra.run.smithi079.stdout:(2/33): ceph-mgr-18.2.1-0.el9.x86_64.rpm 9.0 MB/s | 1.4 MB 00:00 2024-05-04T12:05:54.967 INFO:teuthology.orchestra.run.smithi079.stdout:(3/33): ceph-mds-18.2.1-0.el9.x86_64.rpm 9.5 MB/s | 2.1 MB 00:00 2024-05-04T12:05:54.992 INFO:teuthology.orchestra.run.smithi079.stdout:(4/33): ceph-mgr-modules-core-18.2.1-0.el9.noar 9.6 MB/s | 242 kB 00:00 2024-05-04T12:05:55.097 INFO:teuthology.orchestra.run.smithi079.stdout:(5/33): ceph-mon-18.2.1-0.el9.x86_64.rpm 18 MB/s | 4.4 MB 00:00 2024-05-04T12:05:55.342 INFO:teuthology.orchestra.run.smithi079.stdout:(6/33): ceph-osd-18.2.1-0.el9.x86_64.rpm 41 MB/s | 18 MB 00:00 2024-05-04T12:05:55.501 INFO:teuthology.orchestra.run.smithi079.stdout:(7/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 498 kB/s | 253 kB 00:00 2024-05-04T12:05:55.551 INFO:teuthology.orchestra.run.smithi079.stdout:(8/33): python3-ply-3.11-14.el9.noarch.rpm 511 kB/s | 106 kB 00:00 2024-05-04T12:05:55.578 INFO:teuthology.orchestra.run.smithi079.stdout:(9/33): python3-pycparser-2.20-6.el9.noarch.rpm 1.8 MB/s | 135 kB 00:00 2024-05-04T12:05:55.618 INFO:teuthology.orchestra.run.smithi079.stdout:(10/33): python3-cryptography-36.0.1-4.el9.x86_ 2.4 MB/s | 1.2 MB 00:00 2024-05-04T12:05:55.643 INFO:teuthology.orchestra.run.smithi079.stdout:(11/33): python3-urllib3-1.26.5-5.el9.noarch.rp 3.1 MB/s | 215 kB 00:00 2024-05-04T12:05:55.677 INFO:teuthology.orchestra.run.smithi079.stdout:(12/33): python3-requests-2.25.1-8.el9.noarch.r 992 kB/s | 125 kB 00:00 2024-05-04T12:05:55.911 INFO:teuthology.orchestra.run.smithi079.stdout:(13/33): python3-markupsafe-1.1.1-12.el9.x86_64 130 kB/s | 35 kB 00:00 2024-05-04T12:05:56.019 INFO:teuthology.orchestra.run.smithi079.stdout:(14/33): python3-pytz-2021.1-5.el9.noarch.rpm 149 kB/s | 51 kB 00:00 2024-05-04T12:05:56.095 INFO:teuthology.orchestra.run.smithi079.stdout:(15/33): python3-toml-0.10.2-6.el9.noarch.rpm 227 kB/s | 42 kB 00:00 2024-05-04T12:05:56.120 INFO:teuthology.orchestra.run.smithi079.stdout:(16/33): python3-mako-1.1.4-6.el9.noarch.rpm 343 kB/s | 172 kB 00:00 2024-05-04T12:05:56.345 INFO:teuthology.orchestra.run.smithi079.stdout:(17/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 133 kB/s | 43 kB 00:00 2024-05-04T12:05:56.512 INFO:teuthology.orchestra.run.smithi079.stdout:(18/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 64 kB/s | 11 kB 00:00 2024-05-04T12:05:56.554 INFO:teuthology.orchestra.run.smithi079.stdout:(19/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 374 kB/s | 172 kB 00:00 2024-05-04T12:05:56.621 INFO:teuthology.orchestra.run.smithi079.stdout:(20/33): python3-cherrypy-18.6.1-2.el9.noarch.r 716 kB/s | 358 kB 00:00 2024-05-04T12:05:56.763 INFO:teuthology.orchestra.run.smithi079.stdout:(21/33): python3-jaraco-classes-3.2.1-5.el9.noa 70 kB/s | 18 kB 00:00 2024-05-04T12:05:56.788 INFO:teuthology.orchestra.run.smithi079.stdout:(22/33): python3-jaraco-collections-3.0.0-8.el9 99 kB/s | 23 kB 00:00 2024-05-04T12:05:56.863 INFO:teuthology.orchestra.run.smithi079.stdout:(23/33): python3-jaraco-functools-3.5.0-2.el9.n 80 kB/s | 19 kB 00:00 2024-05-04T12:05:57.030 INFO:teuthology.orchestra.run.smithi079.stdout:(24/33): python3-jaraco-text-3.2.0-6.el9.noarch 73 kB/s | 20 kB 00:00 2024-05-04T12:05:57.114 INFO:teuthology.orchestra.run.smithi079.stdout:(25/33): python3-logutils-0.3.5-21.el9.noarch.r 142 kB/s | 46 kB 00:00 2024-05-04T12:05:57.264 INFO:teuthology.orchestra.run.smithi079.stdout:(26/33): python3-more-itertools-8.12.0-2.el9.no 197 kB/s | 79 kB 00:00 2024-05-04T12:05:57.364 INFO:teuthology.orchestra.run.smithi079.stdout:(27/33): python3-portend-3.1.0-2.el9.noarch.rpm 66 kB/s | 16 kB 00:00 2024-05-04T12:05:57.573 INFO:teuthology.orchestra.run.smithi079.stdout:(28/33): python3-pecan-1.4.2-3.el9.noarch.rpm 502 kB/s | 272 kB 00:00 2024-05-04T12:05:57.615 INFO:teuthology.orchestra.run.smithi079.stdout:(29/33): python3-tempora-5.0.0-2.el9.noarch.rpm 143 kB/s | 36 kB 00:00 2024-05-04T12:05:57.665 INFO:teuthology.orchestra.run.smithi079.stdout:(30/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 224 kB/s | 90 kB 00:00 2024-05-04T12:05:57.907 INFO:teuthology.orchestra.run.smithi079.stdout:(31/33): python3-zc-lockfile-2.0-10.el9.noarch. 83 kB/s | 20 kB 00:00 2024-05-04T12:05:58.057 INFO:teuthology.orchestra.run.smithi079.stdout:(32/33): python3-webob-1.8.7-6.el9.noarch.rpm 475 kB/s | 230 kB 00:00 2024-05-04T12:05:58.157 INFO:teuthology.orchestra.run.smithi079.stdout:(33/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 788 kB/s | 427 kB 00:00 2024-05-04T12:05:58.160 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-05-04T12:05:58.161 INFO:teuthology.orchestra.run.smithi079.stdout:Total 7.9 MB/s | 30 MB 00:03 2024-05-04T12:05:58.279 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-05-04T12:05:58.317 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-05-04T12:05:58.317 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-05-04T12:05:58.603 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-05-04T12:05:58.604 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-05-04T12:05:59.758 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-05-04T12:05:59.818 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-05-04T12:05:59.850 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-05-04T12:05:59.883 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-05-04T12:05:59.974 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-05-04T12:06:00.061 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-05-04T12:06:00.103 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2024-05-04T12:06:00.146 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-05-04T12:06:00.176 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/33 2024-05-04T12:06:00.210 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/33 2024-05-04T12:06:00.244 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/33 2024-05-04T12:06:00.284 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/33 2024-05-04T12:06:00.319 INFO:teuthology.orchestra.run.smithi146.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 1.0 MB/s | 16 MB 00:15 2024-05-04T12:06:00.319 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/33 2024-05-04T12:06:00.320 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-05-04T12:06:00.320 INFO:teuthology.orchestra.run.smithi146.stdout:Total 2.2 MB/s | 38 MB 00:17 2024-05-04T12:06:00.354 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/33 2024-05-04T12:06:00.392 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/33 2024-05-04T12:06:00.464 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 15/33 2024-05-04T12:06:00.610 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 16/33 2024-05-04T12:06:00.671 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-05-04T12:06:00.690 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 17/33 2024-05-04T12:06:00.697 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-05-04T12:06:00.697 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-05-04T12:06:00.745 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 18/33 2024-05-04T12:06:00.785 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 19/33 2024-05-04T12:06:00.843 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-ply-3.11-14.el9.noarch 20/33 2024-05-04T12:06:00.898 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 21/33 2024-05-04T12:06:01.083 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 22/33 2024-05-04T12:06:01.103 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-05-04T12:06:01.104 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-05-04T12:06:01.131 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 23/33 2024-05-04T12:06:01.209 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 24/33 2024-05-04T12:06:01.298 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 25/33 2024-05-04T12:06:01.338 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 26/33 2024-05-04T12:06:01.453 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 27/33 2024-05-04T12:06:01.621 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-mgr-modules-core-2:18.2.1-0.el9.noarch 28/33 2024-05-04T12:06:01.649 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-mgr-2:18.2.1-0.el9.x86_64 29/33 2024-05-04T12:06:01.673 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-mgr-2:18.2.1-0.el9.x86_64 29/33 2024-05-04T12:06:01.673 INFO:teuthology.orchestra.run.smithi079.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-04T12:06:01.673 INFO:teuthology.orchestra.run.smithi079.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-05-04T12:06:01.673 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-04T12:06:01.673 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-04T12:06:01.674 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-05-04T12:06:01.865 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-05-04T12:06:01.923 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-05-04T12:06:01.969 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 2/12 2024-05-04T12:06:02.069 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 3/12 2024-05-04T12:06:02.112 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 4/12 2024-05-04T12:06:02.291 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-05-04T12:06:02.318 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-05-04T12:06:02.388 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-05-04T12:06:03.039 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-05-04T12:06:03.156 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-05-04T12:06:03.175 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-osd-2:18.2.1-0.el9.x86_64 30/33 2024-05-04T12:06:03.201 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-osd-2:18.2.1-0.el9.x86_64 30/33 2024-05-04T12:06:03.201 INFO:teuthology.orchestra.run.smithi079.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-04T12:06:03.202 INFO:teuthology.orchestra.run.smithi079.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-05-04T12:06:03.202 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-04T12:06:03.202 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-04T12:06:03.202 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-05-04T12:06:03.642 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-mon-2:18.2.1-0.el9.x86_64 31/33 2024-05-04T12:06:03.665 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-mon-2:18.2.1-0.el9.x86_64 31/33 2024-05-04T12:06:03.666 INFO:teuthology.orchestra.run.smithi079.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-04T12:06:03.666 INFO:teuthology.orchestra.run.smithi079.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-05-04T12:06:03.666 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-04T12:06:03.666 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-04T12:06:03.666 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-05-04T12:06:03.868 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-mds-2:18.2.1-0.el9.x86_64 32/33 2024-05-04T12:06:03.892 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-mds-2:18.2.1-0.el9.x86_64 32/33 2024-05-04T12:06:03.892 INFO:teuthology.orchestra.run.smithi079.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-04T12:06:03.892 INFO:teuthology.orchestra.run.smithi079.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-05-04T12:06:03.892 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-04T12:06:03.892 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-04T12:06:03.892 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-05-04T12:06:03.955 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-2:18.2.1-0.el9.x86_64 33/33 2024-05-04T12:06:04.390 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-05-04T12:06:05.449 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-05-04T12:06:05.484 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.1-0.el9.noarc 12/12 2024-05-04T12:06:05.534 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-2:18.2.1-0.el9.x86_64 33/33 2024-05-04T12:06:05.534 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-2:18.2.1-0.el9.x86_64 1/33 2024-05-04T12:06:05.534 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-mds-2:18.2.1-0.el9.x86_64 2/33 2024-05-04T12:06:05.534 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-mgr-2:18.2.1-0.el9.x86_64 3/33 2024-05-04T12:06:05.534 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-mon-2:18.2.1-0.el9.x86_64 4/33 2024-05-04T12:06:05.534 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-osd-2:18.2.1-0.el9.x86_64 5/33 2024-05-04T12:06:05.535 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-mgr-modules-core-2:18.2.1-0.el9.noarch 6/33 2024-05-04T12:06:05.535 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/33 2024-05-04T12:06:05.535 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/33 2024-05-04T12:06:05.535 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/33 2024-05-04T12:06:05.535 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/33 2024-05-04T12:06:05.535 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/33 2024-05-04T12:06:05.535 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 12/33 2024-05-04T12:06:05.535 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/33 2024-05-04T12:06:05.536 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/33 2024-05-04T12:06:05.536 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 15/33 2024-05-04T12:06:05.536 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 16/33 2024-05-04T12:06:05.536 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-05-04T12:06:05.536 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-05-04T12:06:05.536 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-05-04T12:06:05.536 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-05-04T12:06:05.536 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-05-04T12:06:05.536 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-05-04T12:06:05.536 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-05-04T12:06:05.536 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-05-04T12:06:05.536 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-05-04T12:06:05.537 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-05-04T12:06:05.537 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-05-04T12:06:05.537 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-05-04T12:06:05.537 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-05-04T12:06:05.537 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-05-04T12:06:05.537 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2024-05-04T12:06:05.537 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-05-04T12:06:06.419 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-05-04T12:06:06.419 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-05-04T12:06:06.419 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-05-04T12:06:06.419 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-2:18.2.1-0.el9.x86_64 2024-05-04T12:06:06.420 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mds-2:18.2.1-0.el9.x86_64 2024-05-04T12:06:06.420 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-2:18.2.1-0.el9.x86_64 2024-05-04T12:06:06.420 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-modules-core-2:18.2.1-0.el9.noarch 2024-05-04T12:06:06.420 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mon-2:18.2.1-0.el9.x86_64 2024-05-04T12:06:06.420 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-osd-2:18.2.1-0.el9.x86_64 2024-05-04T12:06:06.420 INFO:teuthology.orchestra.run.smithi079.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-05-04T12:06:06.420 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-05-04T12:06:06.420 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-05-04T12:06:06.420 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-05-04T12:06:06.420 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-05-04T12:06:06.420 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-05-04T12:06:06.420 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-05-04T12:06:06.420 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-05-04T12:06:06.420 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-05-04T12:06:06.420 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-05-04T12:06:06.421 INFO:teuthology.orchestra.run.smithi079.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-05-04T12:06:06.421 INFO:teuthology.orchestra.run.smithi079.stdout: python3-mako-1.1.4-6.el9.noarch 2024-05-04T12:06:06.421 INFO:teuthology.orchestra.run.smithi079.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-05-04T12:06:06.421 INFO:teuthology.orchestra.run.smithi079.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-05-04T12:06:06.421 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-05-04T12:06:06.421 INFO:teuthology.orchestra.run.smithi079.stdout: python3-ply-3.11-14.el9.noarch 2024-05-04T12:06:06.421 INFO:teuthology.orchestra.run.smithi079.stdout: python3-portend-3.1.0-2.el9.noarch 2024-05-04T12:06:06.421 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-05-04T12:06:06.421 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pycparser-2.20-6.el9.noarch 2024-05-04T12:06:06.421 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pytz-2021.1-5.el9.noarch 2024-05-04T12:06:06.421 INFO:teuthology.orchestra.run.smithi079.stdout: python3-requests-2.25.1-8.el9.noarch 2024-05-04T12:06:06.421 INFO:teuthology.orchestra.run.smithi079.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-05-04T12:06:06.421 INFO:teuthology.orchestra.run.smithi079.stdout: python3-toml-0.10.2-6.el9.noarch 2024-05-04T12:06:06.422 INFO:teuthology.orchestra.run.smithi079.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-05-04T12:06:06.422 INFO:teuthology.orchestra.run.smithi079.stdout: python3-webob-1.8.7-6.el9.noarch 2024-05-04T12:06:06.422 INFO:teuthology.orchestra.run.smithi079.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-05-04T12:06:06.422 INFO:teuthology.orchestra.run.smithi079.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-05-04T12:06:06.422 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-05-04T12:06:06.422 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-05-04T12:06:06.803 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install ceph-base 2024-05-04T12:06:07.310 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:00:53 ago on Sat 04 May 2024 12:05:14 PM UTC. 2024-05-04T12:06:07.387 INFO:teuthology.orchestra.run.smithi079.stdout:Package ceph-base-2:18.2.1-0.el9.x86_64 is already installed. 2024-05-04T12:06:07.428 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-05-04T12:06:07.429 INFO:teuthology.orchestra.run.smithi079.stdout:Nothing to do. 2024-05-04T12:06:07.429 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-05-04T12:06:07.509 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install cephadm 2024-05-04T12:06:08.009 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:00:54 ago on Sat 04 May 2024 12:05:14 PM UTC. 2024-05-04T12:06:08.111 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.1-0.el9.noarc 12/12 2024-05-04T12:06:08.112 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.1-0.el9.noarc 1/12 2024-05-04T12:06:08.112 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-05-04T12:06:08.112 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 3/12 2024-05-04T12:06:08.112 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-05-04T12:06:08.112 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-05-04T12:06:08.112 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-05-04T12:06:08.112 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 7/12 2024-05-04T12:06:08.112 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 8/12 2024-05-04T12:06:08.112 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 9/12 2024-05-04T12:06:08.112 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-05-04T12:06:08.112 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-05-04T12:06:08.128 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-05-04T12:06:08.128 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-05-04T12:06:08.129 INFO:teuthology.orchestra.run.smithi079.stdout: Package Architecture Version Repository Size 2024-05-04T12:06:08.129 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-05-04T12:06:08.129 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-05-04T12:06:08.129 INFO:teuthology.orchestra.run.smithi079.stdout: cephadm noarch 2:18.2.1-0.el9 ceph-noarch 221 k 2024-05-04T12:06:08.129 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-05-04T12:06:08.129 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-05-04T12:06:08.129 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-05-04T12:06:08.129 INFO:teuthology.orchestra.run.smithi079.stdout:Install 1 Package 2024-05-04T12:06:08.129 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-05-04T12:06:08.129 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 221 k 2024-05-04T12:06:08.129 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 213 k 2024-05-04T12:06:08.129 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-05-04T12:06:08.202 INFO:teuthology.orchestra.run.smithi079.stdout:cephadm-18.2.1-0.el9.noarch.rpm 3.0 MB/s | 221 kB 00:00 2024-05-04T12:06:08.202 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-05-04T12:06:08.203 INFO:teuthology.orchestra.run.smithi079.stdout:Total 2.9 MB/s | 221 kB 00:00 2024-05-04T12:06:08.203 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-05-04T12:06:08.207 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-05-04T12:06:08.207 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-05-04T12:06:08.210 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-05-04T12:06:08.210 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-05-04T12:06:08.278 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-05-04T12:06:08.532 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-05-04T12:06:08.532 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-05-04T12:06:08.532 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2024-05-04T12:06:08.532 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-diskprediction-local-2:18.2.1-0.el9.noarch 2024-05-04T12:06:08.532 INFO:teuthology.orchestra.run.smithi146.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-05-04T12:06:08.532 INFO:teuthology.orchestra.run.smithi146.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-05-04T12:06:08.532 INFO:teuthology.orchestra.run.smithi146.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-05-04T12:06:08.532 INFO:teuthology.orchestra.run.smithi146.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-05-04T12:06:08.532 INFO:teuthology.orchestra.run.smithi146.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-05-04T12:06:08.532 INFO:teuthology.orchestra.run.smithi146.stdout: openblas-0.3.21-2.el9.x86_64 2024-05-04T12:06:08.532 INFO:teuthology.orchestra.run.smithi146.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-05-04T12:06:08.532 INFO:teuthology.orchestra.run.smithi146.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-05-04T12:06:08.532 INFO:teuthology.orchestra.run.smithi146.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-05-04T12:06:08.532 INFO:teuthology.orchestra.run.smithi146.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-05-04T12:06:08.533 INFO:teuthology.orchestra.run.smithi146.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-05-04T12:06:08.533 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-05-04T12:06:08.533 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-05-04T12:06:08.755 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: cephadm-2:18.2.1-0.el9.noarch 1/1 2024-05-04T12:06:08.837 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : cephadm-2:18.2.1-0.el9.noarch 1/1 2024-05-04T12:06:08.920 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install ceph-mgr-rook 2024-05-04T12:06:09.163 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: cephadm-2:18.2.1-0.el9.noarch 1/1 2024-05-04T12:06:09.389 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : cephadm-2:18.2.1-0.el9.noarch 1/1 2024-05-04T12:06:09.390 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-05-04T12:06:09.390 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-05-04T12:06:09.390 INFO:teuthology.orchestra.run.smithi079.stdout: cephadm-2:18.2.1-0.el9.noarch 2024-05-04T12:06:09.390 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-05-04T12:06:09.390 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-05-04T12:06:09.454 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:01:23 ago on Sat 04 May 2024 12:04:46 PM UTC. 2024-05-04T12:06:09.585 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-05-04T12:06:09.587 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-05-04T12:06:09.587 INFO:teuthology.orchestra.run.smithi146.stdout: Package Arch Version Repository Size 2024-05-04T12:06:09.587 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-05-04T12:06:09.587 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2024-05-04T12:06:09.587 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-rook noarch 2:18.2.1-0.el9 ceph-noarch 50 k 2024-05-04T12:06:09.587 INFO:teuthology.orchestra.run.smithi146.stdout:Installing dependencies: 2024-05-04T12:06:09.587 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-05-04T12:06:09.587 INFO:teuthology.orchestra.run.smithi146.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-05-04T12:06:09.587 INFO:teuthology.orchestra.run.smithi146.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-05-04T12:06:09.587 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-05-04T12:06:09.587 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-05-04T12:06:09.587 INFO:teuthology.orchestra.run.smithi146.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-05-04T12:06:09.588 INFO:teuthology.orchestra.run.smithi146.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-05-04T12:06:09.588 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-05-04T12:06:09.588 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-05-04T12:06:09.588 INFO:teuthology.orchestra.run.smithi146.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-05-04T12:06:09.588 INFO:teuthology.orchestra.run.smithi146.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-05-04T12:06:09.588 INFO:teuthology.orchestra.run.smithi146.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-05-04T12:06:09.588 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-05-04T12:06:09.588 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-05-04T12:06:09.588 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-05-04T12:06:09.588 INFO:teuthology.orchestra.run.smithi146.stdout:Install 13 Packages 2024-05-04T12:06:09.588 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-05-04T12:06:09.589 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 2.2 M 2024-05-04T12:06:09.589 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 26 M 2024-05-04T12:06:09.589 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-05-04T12:06:09.621 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install ceph-immutable-object-cache 2024-05-04T12:06:09.816 INFO:teuthology.orchestra.run.smithi146.stdout:(1/13): ceph-mgr-rook-18.2.1-0.el9.noarch.rpm 526 kB/s | 50 kB 00:00 2024-05-04T12:06:09.991 INFO:teuthology.orchestra.run.smithi146.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 95 kB/s | 26 kB 00:00 2024-05-04T12:06:10.008 INFO:teuthology.orchestra.run.smithi146.stdout:(3/13): python3-jsonpointer-2.0-4.el9.noarch.rp 67 kB/s | 19 kB 00:00 2024-05-04T12:06:10.130 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:00:56 ago on Sat 04 May 2024 12:05:14 PM UTC. 2024-05-04T12:06:10.250 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-05-04T12:06:10.251 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-05-04T12:06:10.251 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repo Size 2024-05-04T12:06:10.251 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-05-04T12:06:10.251 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-05-04T12:06:10.251 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-immutable-object-cache x86_64 2:18.2.1-0.el9 ceph 142 k 2024-05-04T12:06:10.251 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-05-04T12:06:10.251 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-05-04T12:06:10.251 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-05-04T12:06:10.252 INFO:teuthology.orchestra.run.smithi079.stdout:Install 1 Package 2024-05-04T12:06:10.252 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-05-04T12:06:10.252 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 142 k 2024-05-04T12:06:10.252 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 395 k 2024-05-04T12:06:10.252 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-05-04T12:06:10.317 INFO:teuthology.orchestra.run.smithi146.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 444 kB/s | 222 kB 00:00 2024-05-04T12:06:10.324 INFO:teuthology.orchestra.run.smithi079.stdout:ceph-immutable-object-cache-18.2.1-0.el9.x86_64 1.9 MB/s | 142 kB 00:00 2024-05-04T12:06:10.324 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-05-04T12:06:10.325 INFO:teuthology.orchestra.run.smithi079.stdout:Total 1.9 MB/s | 142 kB 00:00 2024-05-04T12:06:10.325 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-05-04T12:06:10.332 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-05-04T12:06:10.332 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-05-04T12:06:10.342 INFO:teuthology.orchestra.run.smithi146.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 453 kB/s | 159 kB 00:00 2024-05-04T12:06:10.390 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-05-04T12:06:10.391 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-05-04T12:06:10.434 INFO:teuthology.orchestra.run.smithi146.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 459 kB/s | 54 kB 00:00 2024-05-04T12:06:10.468 INFO:teuthology.orchestra.run.smithi146.stdout:(7/13): python3-pyasn1-modules-0.4.8-6.el9.noar 607 kB/s | 279 kB 00:00 2024-05-04T12:06:10.532 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-05-04T12:06:10.609 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-immutable-object-cache-2:18.2.1-0.el9.x86_64 1/1 2024-05-04T12:06:10.633 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.1-0.el9.x86_64 1/1 2024-05-04T12:06:10.633 INFO:teuthology.orchestra.run.smithi079.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-04T12:06:10.633 INFO:teuthology.orchestra.run.smithi079.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-05-04T12:06:10.633 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-05-04T12:06:10.634 INFO:teuthology.orchestra.run.smithi146.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 110 kB/s | 32 kB 00:00 2024-05-04T12:06:10.668 INFO:teuthology.orchestra.run.smithi146.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 64 kB/s | 15 kB 00:00 2024-05-04T12:06:10.885 INFO:teuthology.orchestra.run.smithi146.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 273 kB/s | 59 kB 00:00 2024-05-04T12:06:10.952 INFO:teuthology.orchestra.run.smithi146.stdout:(11/13): python3-google-auth-2.29.0-1.el9.noarc 425 kB/s | 206 kB 00:00 2024-05-04T12:06:11.061 INFO:teuthology.orchestra.run.smithi146.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 512 kB/s | 90 kB 00:00 2024-05-04T12:06:11.169 INFO:teuthology.orchestra.run.smithi146.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 1.9 MB/s | 1.0 MB 00:00 2024-05-04T12:06:11.170 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-05-04T12:06:11.170 INFO:teuthology.orchestra.run.smithi146.stdout:Total 1.4 MB/s | 2.2 MB 00:01 2024-05-04T12:06:11.232 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-05-04T12:06:11.247 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-05-04T12:06:11.247 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-05-04T12:06:11.334 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-05-04T12:06:11.336 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-05-04T12:06:11.523 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-immutable-object-cache-2:18.2.1-0.el9.x86_64 1/1 2024-05-04T12:06:11.523 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-05-04T12:06:11.523 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-05-04T12:06:11.523 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-immutable-object-cache-2:18.2.1-0.el9.x86_64 2024-05-04T12:06:11.523 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-05-04T12:06:11.523 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-05-04T12:06:11.759 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install ceph-mgr 2024-05-04T12:06:11.902 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-05-04T12:06:12.088 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-05-04T12:06:12.146 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-05-04T12:06:12.201 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-05-04T12:06:12.238 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-05-04T12:06:12.266 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:00:58 ago on Sat 04 May 2024 12:05:14 PM UTC. 2024-05-04T12:06:12.280 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-05-04T12:06:12.345 INFO:teuthology.orchestra.run.smithi079.stdout:Package ceph-mgr-2:18.2.1-0.el9.x86_64 is already installed. 2024-05-04T12:06:12.375 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-05-04T12:06:12.386 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-05-04T12:06:12.387 INFO:teuthology.orchestra.run.smithi079.stdout:Nothing to do. 2024-05-04T12:06:12.388 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-05-04T12:06:12.467 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install ceph-mgr-dashboard 2024-05-04T12:06:12.484 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 7/13 2024-05-04T12:06:12.531 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-05-04T12:06:12.971 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:00:58 ago on Sat 04 May 2024 12:05:14 PM UTC. 2024-05-04T12:06:13.090 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-05-04T12:06:13.091 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-05-04T12:06:13.091 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repository Size 2024-05-04T12:06:13.091 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-05-04T12:06:13.091 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-05-04T12:06:13.091 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-dashboard noarch 2:18.2.1-0.el9 ceph-noarch 1.8 M 2024-05-04T12:06:13.091 INFO:teuthology.orchestra.run.smithi079.stdout:Installing dependencies: 2024-05-04T12:06:13.091 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-grafana-dashboards noarch 2:18.2.1-0.el9 ceph-noarch 23 k 2024-05-04T12:06:13.091 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-prometheus-alerts noarch 2:18.2.1-0.el9 ceph-noarch 15 k 2024-05-04T12:06:13.091 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-05-04T12:06:13.091 INFO:teuthology.orchestra.run.smithi079.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-05-04T12:06:13.091 INFO:teuthology.orchestra.run.smithi079.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-05-04T12:06:13.092 INFO:teuthology.orchestra.run.smithi079.stdout:Installing weak dependencies: 2024-05-04T12:06:13.092 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-05-04T12:06:13.092 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-05-04T12:06:13.092 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-05-04T12:06:13.092 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-05-04T12:06:13.092 INFO:teuthology.orchestra.run.smithi079.stdout:Install 7 Packages 2024-05-04T12:06:13.092 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-05-04T12:06:13.093 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 2.1 M 2024-05-04T12:06:13.093 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 8.5 M 2024-05-04T12:06:13.093 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-05-04T12:06:13.269 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-05-04T12:06:13.269 INFO:teuthology.orchestra.run.smithi079.stdout:(1/7): ceph-grafana-dashboards-18.2.1-0.el9.noa 189 kB/s | 23 kB 00:00 2024-05-04T12:06:13.295 INFO:teuthology.orchestra.run.smithi079.stdout:(2/7): ceph-prometheus-alerts-18.2.1-0.el9.noar 99 kB/s | 15 kB 00:00 2024-05-04T12:06:13.319 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-05-04T12:06:13.345 INFO:teuthology.orchestra.run.smithi079.stdout:(3/7): ceph-mgr-dashboard-18.2.1-0.el9.noarch.r 8.9 MB/s | 1.8 MB 00:00 2024-05-04T12:06:13.370 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-05-04T12:06:13.395 INFO:teuthology.orchestra.run.smithi079.stdout:(4/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 72 kB/s | 9.0 kB 00:00 2024-05-04T12:06:13.417 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-05-04T12:06:13.439 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-mgr-rook-2:18.2.1-0.el9.noarch 13/13 2024-05-04T12:06:13.462 INFO:teuthology.orchestra.run.smithi079.stdout:(5/7): python3-jwt-2.4.0-1.el9.noarch.rpm 244 kB/s | 41 kB 00:00 2024-05-04T12:06:13.488 INFO:teuthology.orchestra.run.smithi079.stdout:(6/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 216 kB/s | 31 kB 00:00 2024-05-04T12:06:13.571 INFO:teuthology.orchestra.run.smithi079.stdout:(7/7): python3-routes-2.5.1-5.el9.noarch.rpm 1.0 MB/s | 188 kB 00:00 2024-05-04T12:06:13.572 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-05-04T12:06:13.573 INFO:teuthology.orchestra.run.smithi079.stdout:Total 4.3 MB/s | 2.1 MB 00:00 2024-05-04T12:06:13.573 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-05-04T12:06:13.581 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-05-04T12:06:13.581 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-05-04T12:06:13.633 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-05-04T12:06:13.633 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-05-04T12:06:13.943 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-05-04T12:06:14.005 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-05-04T12:06:14.039 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-05-04T12:06:14.085 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-05-04T12:06:14.118 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-05-04T12:06:14.143 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.1-0.el9.noarch 13/13 2024-05-04T12:06:14.143 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-mgr-rook-2:18.2.1-0.el9.noarch 1/13 2024-05-04T12:06:14.144 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-05-04T12:06:14.144 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-05-04T12:06:14.144 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-05-04T12:06:14.144 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-05-04T12:06:14.144 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-05-04T12:06:14.144 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-05-04T12:06:14.144 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-05-04T12:06:14.144 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-05-04T12:06:14.144 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 10/13 2024-05-04T12:06:14.144 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-05-04T12:06:14.144 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-05-04T12:06:14.147 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-prometheus-alerts-2:18.2.1-0.el9.noarch 5/7 2024-05-04T12:06:14.382 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-grafana-dashboards-2:18.2.1-0.el9.noarch 6/7 2024-05-04T12:06:14.416 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-mgr-dashboard-2:18.2.1-0.el9.noarch 7/7 2024-05-04T12:06:14.472 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-05-04T12:06:14.472 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-05-04T12:06:14.472 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2024-05-04T12:06:14.472 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-rook-2:18.2.1-0.el9.noarch 2024-05-04T12:06:14.472 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-05-04T12:06:14.472 INFO:teuthology.orchestra.run.smithi146.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-05-04T12:06:14.472 INFO:teuthology.orchestra.run.smithi146.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-05-04T12:06:14.472 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-05-04T12:06:14.472 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-05-04T12:06:14.472 INFO:teuthology.orchestra.run.smithi146.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-05-04T12:06:14.473 INFO:teuthology.orchestra.run.smithi146.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-05-04T12:06:14.473 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-05-04T12:06:14.473 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-05-04T12:06:14.473 INFO:teuthology.orchestra.run.smithi146.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-05-04T12:06:14.473 INFO:teuthology.orchestra.run.smithi146.stdout: python3-rsa-4.9-2.el9.noarch 2024-05-04T12:06:14.473 INFO:teuthology.orchestra.run.smithi146.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-05-04T12:06:14.473 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-05-04T12:06:14.473 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-05-04T12:06:14.724 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install ceph-mgr-cephadm 2024-05-04T12:06:14.883 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.1-0.el9.noarch 7/7 2024-05-04T12:06:14.883 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-grafana-dashboards-2:18.2.1-0.el9.noarch 1/7 2024-05-04T12:06:14.883 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-mgr-dashboard-2:18.2.1-0.el9.noarch 2/7 2024-05-04T12:06:14.883 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-prometheus-alerts-2:18.2.1-0.el9.noarch 3/7 2024-05-04T12:06:14.883 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-05-04T12:06:14.883 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-05-04T12:06:14.884 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-05-04T12:06:15.130 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-05-04T12:06:15.130 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-05-04T12:06:15.131 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-05-04T12:06:15.131 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-grafana-dashboards-2:18.2.1-0.el9.noarch 2024-05-04T12:06:15.131 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-dashboard-2:18.2.1-0.el9.noarch 2024-05-04T12:06:15.131 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-prometheus-alerts-2:18.2.1-0.el9.noarch 2024-05-04T12:06:15.131 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-05-04T12:06:15.131 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-05-04T12:06:15.131 INFO:teuthology.orchestra.run.smithi079.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-05-04T12:06:15.131 INFO:teuthology.orchestra.run.smithi079.stdout: python3-routes-2.5.1-5.el9.noarch 2024-05-04T12:06:15.131 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-05-04T12:06:15.131 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-05-04T12:06:15.236 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:01:29 ago on Sat 04 May 2024 12:04:46 PM UTC. 2024-05-04T12:06:15.355 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-05-04T12:06:15.356 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-05-04T12:06:15.356 INFO:teuthology.orchestra.run.smithi146.stdout: Package Arch Version Repository Size 2024-05-04T12:06:15.356 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-05-04T12:06:15.356 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2024-05-04T12:06:15.356 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-cephadm noarch 2:18.2.1-0.el9 ceph-noarch 132 k 2024-05-04T12:06:15.356 INFO:teuthology.orchestra.run.smithi146.stdout:Installing dependencies: 2024-05-04T12:06:15.356 INFO:teuthology.orchestra.run.smithi146.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-05-04T12:06:15.356 INFO:teuthology.orchestra.run.smithi146.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-05-04T12:06:15.357 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-05-04T12:06:15.357 INFO:teuthology.orchestra.run.smithi146.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-05-04T12:06:15.357 INFO:teuthology.orchestra.run.smithi146.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-05-04T12:06:15.357 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-05-04T12:06:15.357 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-05-04T12:06:15.357 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-05-04T12:06:15.357 INFO:teuthology.orchestra.run.smithi146.stdout:Install 6 Packages 2024-05-04T12:06:15.357 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-05-04T12:06:15.358 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 7.0 M 2024-05-04T12:06:15.358 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 33 M 2024-05-04T12:06:15.358 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-05-04T12:06:15.380 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install ceph-mgr-diskprediction-local 2024-05-04T12:06:15.562 INFO:teuthology.orchestra.run.smithi146.stdout:(1/6): ceph-mgr-cephadm-18.2.1-0.el9.noarch.rpm 1.5 MB/s | 132 kB 00:00 2024-05-04T12:06:15.629 INFO:teuthology.orchestra.run.smithi146.stdout:(2/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.6 MB/s | 248 kB 00:00 2024-05-04T12:06:15.813 INFO:teuthology.orchestra.run.smithi146.stdout:(3/6): python3-babel-2.9.1-2.el9.noarch.rpm 18 MB/s | 6.0 MB 00:00 2024-05-04T12:06:15.887 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:01 ago on Sat 04 May 2024 12:05:14 PM UTC. 2024-05-04T12:06:16.008 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-05-04T12:06:16.009 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-05-04T12:06:16.009 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repository Size 2024-05-04T12:06:16.010 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-05-04T12:06:16.010 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-05-04T12:06:16.010 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.1-0.el9 ceph-noarch 7.4 M 2024-05-04T12:06:16.010 INFO:teuthology.orchestra.run.smithi079.stdout:Installing dependencies: 2024-05-04T12:06:16.010 INFO:teuthology.orchestra.run.smithi079.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-05-04T12:06:16.010 INFO:teuthology.orchestra.run.smithi079.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-05-04T12:06:16.010 INFO:teuthology.orchestra.run.smithi079.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-05-04T12:06:16.010 INFO:teuthology.orchestra.run.smithi079.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-05-04T12:06:16.010 INFO:teuthology.orchestra.run.smithi079.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-05-04T12:06:16.010 INFO:teuthology.orchestra.run.smithi079.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-05-04T12:06:16.010 INFO:teuthology.orchestra.run.smithi079.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-05-04T12:06:16.010 INFO:teuthology.orchestra.run.smithi079.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-05-04T12:06:16.010 INFO:teuthology.orchestra.run.smithi079.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-05-04T12:06:16.010 INFO:teuthology.orchestra.run.smithi079.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-05-04T12:06:16.011 INFO:teuthology.orchestra.run.smithi079.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-05-04T12:06:16.011 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-05-04T12:06:16.011 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-05-04T12:06:16.011 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-05-04T12:06:16.011 INFO:teuthology.orchestra.run.smithi079.stdout:Install 12 Packages 2024-05-04T12:06:16.011 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-05-04T12:06:16.011 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 38 M 2024-05-04T12:06:16.012 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 209 M 2024-05-04T12:06:16.012 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-05-04T12:06:16.038 INFO:teuthology.orchestra.run.smithi146.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 150 kB/s | 58 kB 00:00 2024-05-04T12:06:16.188 INFO:teuthology.orchestra.run.smithi146.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 136 kB/s | 51 kB 00:00 2024-05-04T12:06:16.213 INFO:teuthology.orchestra.run.smithi146.stdout:(6/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 842 kB/s | 548 kB 00:00 2024-05-04T12:06:16.216 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-05-04T12:06:16.216 INFO:teuthology.orchestra.run.smithi146.stdout:Total 8.1 MB/s | 7.0 MB 00:00 2024-05-04T12:06:16.286 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-05-04T12:06:16.297 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-05-04T12:06:16.298 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-05-04T12:06:16.394 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-05-04T12:06:16.395 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-05-04T12:06:16.402 INFO:teuthology.orchestra.run.smithi079.stdout:(1/12): ceph-mgr-diskprediction-local-18.2.1-0. 46 MB/s | 7.4 MB 00:00 2024-05-04T12:06:16.693 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-05-04T12:06:16.864 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-05-04T12:06:16.915 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-05-04T12:06:17.019 INFO:teuthology.orchestra.run.smithi079.stdout:(2/12): flexiblas-3.0.4-8.el9.x86_64.rpm 53 kB/s | 33 kB 00:00 2024-05-04T12:06:17.136 INFO:teuthology.orchestra.run.smithi079.stdout:(3/12): libquadmath-11.4.1-3.el9.x86_64.rpm 218 kB/s | 195 kB 00:00 2024-05-04T12:06:17.311 INFO:teuthology.orchestra.run.smithi079.stdout:(4/12): libgfortran-11.4.1-3.el9.x86_64.rpm 750 kB/s | 803 kB 00:01 2024-05-04T12:06:17.439 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-05-04T12:06:17.518 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-05-04T12:06:17.578 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-05-04T12:06:17.602 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-mgr-cephadm-2:18.2.1-0.el9.noarch 6/6 2024-05-04T12:06:17.679 INFO:teuthology.orchestra.run.smithi079.stdout:(5/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 34 kB/s | 18 kB 00:00 2024-05-04T12:06:17.954 INFO:teuthology.orchestra.run.smithi079.stdout:(6/12): openblas-0.3.21-2.el9.x86_64.rpm 55 kB/s | 35 kB 00:00 2024-05-04T12:06:18.154 INFO:teuthology.orchestra.run.smithi079.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 2.6 MB/s | 3.0 MB 00:01 2024-05-04T12:06:18.356 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.1-0.el9.noarch 6/6 2024-05-04T12:06:18.356 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-mgr-cephadm-2:18.2.1-0.el9.noarch 1/6 2024-05-04T12:06:18.356 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-05-04T12:06:18.356 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-05-04T12:06:18.356 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-05-04T12:06:18.356 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-05-04T12:06:18.611 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-05-04T12:06:18.611 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-05-04T12:06:18.611 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2024-05-04T12:06:18.611 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-cephadm-2:18.2.1-0.el9.noarch 2024-05-04T12:06:18.611 INFO:teuthology.orchestra.run.smithi146.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-05-04T12:06:18.611 INFO:teuthology.orchestra.run.smithi146.stdout: python3-babel-2.9.1-2.el9.noarch 2024-05-04T12:06:18.611 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-05-04T12:06:18.612 INFO:teuthology.orchestra.run.smithi146.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-05-04T12:06:18.612 INFO:teuthology.orchestra.run.smithi146.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-05-04T12:06:18.612 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-05-04T12:06:18.612 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-05-04T12:06:18.846 INFO:teuthology.orchestra.run.smithi079.stdout:(8/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 4.1 MB/s | 4.7 MB 00:01 2024-05-04T12:06:18.854 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install ceph-fuse 2024-05-04T12:06:18.871 INFO:teuthology.orchestra.run.smithi079.stdout:(9/12): python3-devel-3.9.18-3.el9.x86_64.rpm 266 kB/s | 244 kB 00:00 2024-05-04T12:06:19.346 INFO:teuthology.orchestra.run.smithi079.stdout:(10/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 4.3 MB/s | 5.1 MB 00:01 2024-05-04T12:06:19.368 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:01:33 ago on Sat 04 May 2024 12:04:46 PM UTC. 2024-05-04T12:06:19.488 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-05-04T12:06:19.488 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-05-04T12:06:19.488 INFO:teuthology.orchestra.run.smithi146.stdout: Package Architecture Version Repository Size 2024-05-04T12:06:19.488 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-05-04T12:06:19.488 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2024-05-04T12:06:19.488 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-fuse x86_64 2:18.2.1-0.el9 ceph 839 k 2024-05-04T12:06:19.488 INFO:teuthology.orchestra.run.smithi146.stdout:Installing dependencies: 2024-05-04T12:06:19.489 INFO:teuthology.orchestra.run.smithi146.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-05-04T12:06:19.489 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-05-04T12:06:19.489 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-05-04T12:06:19.489 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-05-04T12:06:19.489 INFO:teuthology.orchestra.run.smithi146.stdout:Install 2 Packages 2024-05-04T12:06:19.489 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-05-04T12:06:19.489 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 920 k 2024-05-04T12:06:19.490 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 2.7 M 2024-05-04T12:06:19.490 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-05-04T12:06:19.671 INFO:teuthology.orchestra.run.smithi079.stdout:(11/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 304 kB/s | 250 kB 00:00 2024-05-04T12:06:19.697 INFO:teuthology.orchestra.run.smithi146.stdout:(1/2): ceph-fuse-18.2.1-0.el9.x86_64.rpm 9.0 MB/s | 839 kB 00:00 2024-05-04T12:06:20.306 INFO:teuthology.orchestra.run.smithi146.stdout:(2/2): fuse-2.9.9-15.el9.x86_64.rpm 115 kB/s | 80 kB 00:00 2024-05-04T12:06:20.307 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-05-04T12:06:20.307 INFO:teuthology.orchestra.run.smithi146.stdout:Total 1.1 MB/s | 920 kB 00:00 2024-05-04T12:06:20.319 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-05-04T12:06:20.342 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-05-04T12:06:20.342 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-05-04T12:06:20.355 INFO:teuthology.orchestra.run.smithi079.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 11 MB/s | 16 MB 00:01 2024-05-04T12:06:20.356 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-05-04T12:06:20.356 INFO:teuthology.orchestra.run.smithi079.stdout:Total 8.8 MB/s | 38 MB 00:04 2024-05-04T12:06:20.419 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-05-04T12:06:20.420 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-05-04T12:06:20.623 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-05-04T12:06:20.702 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-05-04T12:06:20.728 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-05-04T12:06:20.728 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-05-04T12:06:20.753 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-05-04T12:06:20.820 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-fuse-2:18.2.1-0.el9.x86_64 2/2 2024-05-04T12:06:21.134 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-05-04T12:06:21.135 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-05-04T12:06:21.488 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-fuse-2:18.2.1-0.el9.x86_64 2/2 2024-05-04T12:06:21.488 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-fuse-2:18.2.1-0.el9.x86_64 1/2 2024-05-04T12:06:21.746 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-05-04T12:06:21.746 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-05-04T12:06:21.746 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2024-05-04T12:06:21.747 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-fuse-2:18.2.1-0.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-05-04T12:06:21.747 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-05-04T12:06:21.747 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-05-04T12:06:21.895 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-05-04T12:06:21.959 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-05-04T12:06:21.980 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install librados-devel 2024-05-04T12:06:22.003 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 2/12 2024-05-04T12:06:22.107 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 3/12 2024-05-04T12:06:22.138 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 4/12 2024-05-04T12:06:22.316 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-05-04T12:06:22.345 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-05-04T12:06:22.422 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-05-04T12:06:22.504 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:01:36 ago on Sat 04 May 2024 12:04:46 PM UTC. 2024-05-04T12:06:22.625 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-05-04T12:06:22.625 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-05-04T12:06:22.625 INFO:teuthology.orchestra.run.smithi146.stdout: Package Architecture Version Repository Size 2024-05-04T12:06:22.626 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-05-04T12:06:22.626 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2024-05-04T12:06:22.626 INFO:teuthology.orchestra.run.smithi146.stdout: librados-devel x86_64 2:18.2.1-0.el9 ceph 127 k 2024-05-04T12:06:22.626 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-05-04T12:06:22.626 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-05-04T12:06:22.626 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-05-04T12:06:22.626 INFO:teuthology.orchestra.run.smithi146.stdout:Install 1 Package 2024-05-04T12:06:22.626 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-05-04T12:06:22.627 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 127 k 2024-05-04T12:06:22.629 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 456 k 2024-05-04T12:06:22.629 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-05-04T12:06:22.710 INFO:teuthology.orchestra.run.smithi146.stdout:librados-devel-18.2.1-0.el9.x86_64.rpm 1.5 MB/s | 127 kB 00:00 2024-05-04T12:06:22.710 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-05-04T12:06:22.711 INFO:teuthology.orchestra.run.smithi146.stdout:Total 1.5 MB/s | 127 kB 00:00 2024-05-04T12:06:22.711 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-05-04T12:06:22.717 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-05-04T12:06:22.717 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-05-04T12:06:22.780 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-05-04T12:06:22.780 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-05-04T12:06:22.934 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-05-04T12:06:23.048 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : librados-devel-2:18.2.1-0.el9.x86_64 1/1 2024-05-04T12:06:23.103 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-05-04T12:06:23.218 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-05-04T12:06:23.435 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: librados-devel-2:18.2.1-0.el9.x86_64 1/1 2024-05-04T12:06:23.670 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : librados-devel-2:18.2.1-0.el9.x86_64 1/1 2024-05-04T12:06:23.670 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-05-04T12:06:23.670 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2024-05-04T12:06:23.670 INFO:teuthology.orchestra.run.smithi146.stdout: librados-devel-2:18.2.1-0.el9.x86_64 2024-05-04T12:06:23.670 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-05-04T12:06:23.670 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-05-04T12:06:23.886 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install libcephfs2 2024-05-04T12:06:24.416 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:01:38 ago on Sat 04 May 2024 12:04:46 PM UTC. 2024-05-04T12:06:24.470 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-05-04T12:06:24.496 INFO:teuthology.orchestra.run.smithi146.stdout:Package libcephfs2-2:18.2.1-0.el9.x86_64 is already installed. 2024-05-04T12:06:24.539 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-05-04T12:06:24.539 INFO:teuthology.orchestra.run.smithi146.stdout:Nothing to do. 2024-05-04T12:06:24.540 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-05-04T12:06:24.602 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install libcephfs-devel 2024-05-04T12:06:25.121 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:01:39 ago on Sat 04 May 2024 12:04:46 PM UTC. 2024-05-04T12:06:25.242 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-05-04T12:06:25.243 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-05-04T12:06:25.243 INFO:teuthology.orchestra.run.smithi146.stdout: Package Architecture Version Repository Size 2024-05-04T12:06:25.243 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-05-04T12:06:25.243 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2024-05-04T12:06:25.243 INFO:teuthology.orchestra.run.smithi146.stdout: libcephfs-devel x86_64 2:18.2.1-0.el9 ceph 31 k 2024-05-04T12:06:25.243 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-05-04T12:06:25.243 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-05-04T12:06:25.243 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-05-04T12:06:25.243 INFO:teuthology.orchestra.run.smithi146.stdout:Install 1 Package 2024-05-04T12:06:25.244 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-05-04T12:06:25.244 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 31 k 2024-05-04T12:06:25.244 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 139 k 2024-05-04T12:06:25.244 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-05-04T12:06:25.322 INFO:teuthology.orchestra.run.smithi146.stdout:libcephfs-devel-18.2.1-0.el9.x86_64.rpm 398 kB/s | 31 kB 00:00 2024-05-04T12:06:25.322 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-05-04T12:06:25.322 INFO:teuthology.orchestra.run.smithi146.stdout:Total 392 kB/s | 31 kB 00:00 2024-05-04T12:06:25.322 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-05-04T12:06:25.326 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-05-04T12:06:25.326 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-05-04T12:06:25.328 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-05-04T12:06:25.328 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-05-04T12:06:25.416 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-05-04T12:06:25.490 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-05-04T12:06:25.525 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libcephfs-devel-2:18.2.1-0.el9.x86_64 1/1 2024-05-04T12:06:25.527 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.1-0.el9.noarc 12/12 2024-05-04T12:06:25.862 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: libcephfs-devel-2:18.2.1-0.el9.x86_64 1/1 2024-05-04T12:06:26.076 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libcephfs-devel-2:18.2.1-0.el9.x86_64 1/1 2024-05-04T12:06:26.076 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-05-04T12:06:26.077 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2024-05-04T12:06:26.077 INFO:teuthology.orchestra.run.smithi146.stdout: libcephfs-devel-2:18.2.1-0.el9.x86_64 2024-05-04T12:06:26.077 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-05-04T12:06:26.077 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-05-04T12:06:26.236 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install librados2 2024-05-04T12:06:26.750 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:01:40 ago on Sat 04 May 2024 12:04:46 PM UTC. 2024-05-04T12:06:26.829 INFO:teuthology.orchestra.run.smithi146.stdout:Package librados2-2:18.2.1-0.el9.x86_64 is already installed. 2024-05-04T12:06:26.871 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-05-04T12:06:26.872 INFO:teuthology.orchestra.run.smithi146.stdout:Nothing to do. 2024-05-04T12:06:26.872 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-05-04T12:06:26.937 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install librbd1 2024-05-04T12:06:27.447 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:01:41 ago on Sat 04 May 2024 12:04:46 PM UTC. 2024-05-04T12:06:27.526 INFO:teuthology.orchestra.run.smithi146.stdout:Package librbd1-2:18.2.1-0.el9.x86_64 is already installed. 2024-05-04T12:06:27.566 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-05-04T12:06:27.567 INFO:teuthology.orchestra.run.smithi146.stdout:Nothing to do. 2024-05-04T12:06:27.567 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-05-04T12:06:27.631 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install python3-rados 2024-05-04T12:06:28.141 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:01:42 ago on Sat 04 May 2024 12:04:46 PM UTC. 2024-05-04T12:06:28.220 INFO:teuthology.orchestra.run.smithi146.stdout:Package python3-rados-2:18.2.1-0.el9.x86_64 is already installed. 2024-05-04T12:06:28.261 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-05-04T12:06:28.261 INFO:teuthology.orchestra.run.smithi146.stdout:Nothing to do. 2024-05-04T12:06:28.261 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-05-04T12:06:28.328 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install python3-rgw 2024-05-04T12:06:28.423 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.1-0.el9.noarc 12/12 2024-05-04T12:06:28.423 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.1-0.el9.noarc 1/12 2024-05-04T12:06:28.423 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-05-04T12:06:28.423 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 3/12 2024-05-04T12:06:28.423 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-05-04T12:06:28.424 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-05-04T12:06:28.424 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-05-04T12:06:28.424 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 7/12 2024-05-04T12:06:28.424 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 8/12 2024-05-04T12:06:28.424 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 9/12 2024-05-04T12:06:28.424 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-05-04T12:06:28.424 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-05-04T12:06:28.836 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-05-04T12:06:28.836 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-05-04T12:06:28.836 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-05-04T12:06:28.836 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-diskprediction-local-2:18.2.1-0.el9.noarch 2024-05-04T12:06:28.836 INFO:teuthology.orchestra.run.smithi079.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-05-04T12:06:28.836 INFO:teuthology.orchestra.run.smithi079.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-05-04T12:06:28.837 INFO:teuthology.orchestra.run.smithi079.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-05-04T12:06:28.837 INFO:teuthology.orchestra.run.smithi079.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-05-04T12:06:28.837 INFO:teuthology.orchestra.run.smithi079.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-05-04T12:06:28.837 INFO:teuthology.orchestra.run.smithi079.stdout: openblas-0.3.21-2.el9.x86_64 2024-05-04T12:06:28.837 INFO:teuthology.orchestra.run.smithi079.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-05-04T12:06:28.837 INFO:teuthology.orchestra.run.smithi079.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-05-04T12:06:28.837 INFO:teuthology.orchestra.run.smithi079.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-05-04T12:06:28.837 INFO:teuthology.orchestra.run.smithi079.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-05-04T12:06:28.837 INFO:teuthology.orchestra.run.smithi079.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-05-04T12:06:28.837 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-05-04T12:06:28.837 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-05-04T12:06:28.843 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:01:42 ago on Sat 04 May 2024 12:04:46 PM UTC. 2024-05-04T12:06:28.922 INFO:teuthology.orchestra.run.smithi146.stdout:Package python3-rgw-2:18.2.1-0.el9.x86_64 is already installed. 2024-05-04T12:06:28.962 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-05-04T12:06:28.963 INFO:teuthology.orchestra.run.smithi146.stdout:Nothing to do. 2024-05-04T12:06:28.963 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-05-04T12:06:29.029 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install python3-cephfs 2024-05-04T12:06:29.232 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install ceph-mgr-rook 2024-05-04T12:06:29.545 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:01:43 ago on Sat 04 May 2024 12:04:46 PM UTC. 2024-05-04T12:06:29.625 INFO:teuthology.orchestra.run.smithi146.stdout:Package python3-cephfs-2:18.2.1-0.el9.x86_64 is already installed. 2024-05-04T12:06:29.665 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-05-04T12:06:29.666 INFO:teuthology.orchestra.run.smithi146.stdout:Nothing to do. 2024-05-04T12:06:29.666 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-05-04T12:06:29.726 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install python3-rbd 2024-05-04T12:06:29.749 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:15 ago on Sat 04 May 2024 12:05:14 PM UTC. 2024-05-04T12:06:29.870 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-05-04T12:06:29.871 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-05-04T12:06:29.871 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repository Size 2024-05-04T12:06:29.871 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-05-04T12:06:29.871 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-05-04T12:06:29.871 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-rook noarch 2:18.2.1-0.el9 ceph-noarch 50 k 2024-05-04T12:06:29.872 INFO:teuthology.orchestra.run.smithi079.stdout:Installing dependencies: 2024-05-04T12:06:29.872 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-05-04T12:06:29.872 INFO:teuthology.orchestra.run.smithi079.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-05-04T12:06:29.872 INFO:teuthology.orchestra.run.smithi079.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-05-04T12:06:29.872 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-05-04T12:06:29.872 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-05-04T12:06:29.872 INFO:teuthology.orchestra.run.smithi079.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-05-04T12:06:29.872 INFO:teuthology.orchestra.run.smithi079.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-05-04T12:06:29.872 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-05-04T12:06:29.872 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-05-04T12:06:29.872 INFO:teuthology.orchestra.run.smithi079.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-05-04T12:06:29.872 INFO:teuthology.orchestra.run.smithi079.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-05-04T12:06:29.872 INFO:teuthology.orchestra.run.smithi079.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-05-04T12:06:29.873 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-05-04T12:06:29.873 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-05-04T12:06:29.873 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-05-04T12:06:29.873 INFO:teuthology.orchestra.run.smithi079.stdout:Install 13 Packages 2024-05-04T12:06:29.873 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-05-04T12:06:29.873 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 2.2 M 2024-05-04T12:06:29.874 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 26 M 2024-05-04T12:06:29.874 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-05-04T12:06:30.070 INFO:teuthology.orchestra.run.smithi079.stdout:(1/13): ceph-mgr-rook-18.2.1-0.el9.noarch.rpm 856 kB/s | 50 kB 00:00 2024-05-04T12:06:30.235 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:01:44 ago on Sat 04 May 2024 12:04:46 PM UTC. 2024-05-04T12:06:30.313 INFO:teuthology.orchestra.run.smithi146.stdout:Package python3-rbd-2:18.2.1-0.el9.x86_64 is already installed. 2024-05-04T12:06:30.354 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-05-04T12:06:30.354 INFO:teuthology.orchestra.run.smithi146.stdout:Nothing to do. 2024-05-04T12:06:30.355 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-05-04T12:06:30.425 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install rbd-fuse 2024-05-04T12:06:30.940 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:01:44 ago on Sat 04 May 2024 12:04:46 PM UTC. 2024-05-04T12:06:31.059 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-05-04T12:06:31.060 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-05-04T12:06:31.060 INFO:teuthology.orchestra.run.smithi146.stdout: Package Architecture Version Repository Size 2024-05-04T12:06:31.060 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-05-04T12:06:31.060 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2024-05-04T12:06:31.060 INFO:teuthology.orchestra.run.smithi146.stdout: rbd-fuse x86_64 2:18.2.1-0.el9 ceph 86 k 2024-05-04T12:06:31.060 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-05-04T12:06:31.060 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-05-04T12:06:31.060 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-05-04T12:06:31.060 INFO:teuthology.orchestra.run.smithi146.stdout:Install 1 Package 2024-05-04T12:06:31.061 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-05-04T12:06:31.061 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 86 k 2024-05-04T12:06:31.061 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 226 k 2024-05-04T12:06:31.061 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-05-04T12:06:31.141 INFO:teuthology.orchestra.run.smithi146.stdout:rbd-fuse-18.2.1-0.el9.x86_64.rpm 1.0 MB/s | 86 kB 00:00 2024-05-04T12:06:31.142 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-05-04T12:06:31.142 INFO:teuthology.orchestra.run.smithi146.stdout:Total 1.0 MB/s | 86 kB 00:00 2024-05-04T12:06:31.142 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-05-04T12:06:31.149 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-05-04T12:06:31.149 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-05-04T12:06:31.210 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-05-04T12:06:31.211 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-05-04T12:06:31.370 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-05-04T12:06:31.478 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : rbd-fuse-2:18.2.1-0.el9.x86_64 1/1 2024-05-04T12:06:31.794 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: rbd-fuse-2:18.2.1-0.el9.x86_64 1/1 2024-05-04T12:06:32.034 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : rbd-fuse-2:18.2.1-0.el9.x86_64 1/1 2024-05-04T12:06:32.034 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-05-04T12:06:32.035 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2024-05-04T12:06:32.035 INFO:teuthology.orchestra.run.smithi146.stdout: rbd-fuse-2:18.2.1-0.el9.x86_64 2024-05-04T12:06:32.035 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-05-04T12:06:32.035 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-05-04T12:06:32.307 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install rbd-mirror 2024-05-04T12:06:32.829 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:01:46 ago on Sat 04 May 2024 12:04:46 PM UTC. 2024-05-04T12:06:32.950 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-05-04T12:06:32.951 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-05-04T12:06:32.951 INFO:teuthology.orchestra.run.smithi146.stdout: Package Architecture Version Repository Size 2024-05-04T12:06:32.951 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-05-04T12:06:32.952 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2024-05-04T12:06:32.952 INFO:teuthology.orchestra.run.smithi146.stdout: rbd-mirror x86_64 2:18.2.1-0.el9 ceph 3.0 M 2024-05-04T12:06:32.952 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-05-04T12:06:32.952 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-05-04T12:06:32.952 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-05-04T12:06:32.952 INFO:teuthology.orchestra.run.smithi146.stdout:Install 1 Package 2024-05-04T12:06:32.952 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-05-04T12:06:32.952 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 3.0 M 2024-05-04T12:06:32.952 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 12 M 2024-05-04T12:06:32.952 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-05-04T12:06:33.087 INFO:teuthology.orchestra.run.smithi146.stdout:rbd-mirror-18.2.1-0.el9.x86_64.rpm 22 MB/s | 3.0 MB 00:00 2024-05-04T12:06:33.088 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-05-04T12:06:33.088 INFO:teuthology.orchestra.run.smithi146.stdout:Total 22 MB/s | 3.0 MB 00:00 2024-05-04T12:06:33.088 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-05-04T12:06:33.097 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-05-04T12:06:33.097 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-05-04T12:06:33.179 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-05-04T12:06:33.179 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-05-04T12:06:33.591 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-05-04T12:06:33.654 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : rbd-mirror-2:18.2.1-0.el9.x86_64 1/1 2024-05-04T12:06:33.679 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: rbd-mirror-2:18.2.1-0.el9.x86_64 1/1 2024-05-04T12:06:33.679 INFO:teuthology.orchestra.run.smithi146.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-04T12:06:33.679 INFO:teuthology.orchestra.run.smithi146.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-05-04T12:06:33.679 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-04T12:06:33.679 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-04T12:06:33.679 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-05-04T12:06:34.701 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : rbd-mirror-2:18.2.1-0.el9.x86_64 1/1 2024-05-04T12:06:34.701 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-05-04T12:06:34.701 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2024-05-04T12:06:34.701 INFO:teuthology.orchestra.run.smithi146.stdout: rbd-mirror-2:18.2.1-0.el9.x86_64 2024-05-04T12:06:34.702 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-05-04T12:06:34.702 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-05-04T12:06:34.948 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install rbd-nbd 2024-05-04T12:06:35.469 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:01:49 ago on Sat 04 May 2024 12:04:46 PM UTC. 2024-05-04T12:06:35.594 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-05-04T12:06:35.595 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-05-04T12:06:35.595 INFO:teuthology.orchestra.run.smithi146.stdout: Package Architecture Version Repository Size 2024-05-04T12:06:35.595 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-05-04T12:06:35.595 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2024-05-04T12:06:35.595 INFO:teuthology.orchestra.run.smithi146.stdout: rbd-nbd x86_64 2:18.2.1-0.el9 ceph 171 k 2024-05-04T12:06:35.595 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-05-04T12:06:35.595 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-05-04T12:06:35.595 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-05-04T12:06:35.596 INFO:teuthology.orchestra.run.smithi146.stdout:Install 1 Package 2024-05-04T12:06:35.596 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-05-04T12:06:35.596 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 171 k 2024-05-04T12:06:35.596 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 494 k 2024-05-04T12:06:35.596 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-05-04T12:06:35.673 INFO:teuthology.orchestra.run.smithi146.stdout:rbd-nbd-18.2.1-0.el9.x86_64.rpm 2.2 MB/s | 171 kB 00:00 2024-05-04T12:06:35.674 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-05-04T12:06:35.674 INFO:teuthology.orchestra.run.smithi146.stdout:Total 2.1 MB/s | 171 kB 00:00 2024-05-04T12:06:35.674 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-05-04T12:06:35.682 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-05-04T12:06:35.682 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-05-04T12:06:35.745 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-05-04T12:06:35.746 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-05-04T12:06:35.896 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-05-04T12:06:36.010 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : rbd-nbd-2:18.2.1-0.el9.x86_64 1/1 2024-05-04T12:06:36.401 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: rbd-nbd-2:18.2.1-0.el9.x86_64 1/1 2024-05-04T12:06:36.633 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : rbd-nbd-2:18.2.1-0.el9.x86_64 1/1 2024-05-04T12:06:36.633 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-05-04T12:06:36.634 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2024-05-04T12:06:36.634 INFO:teuthology.orchestra.run.smithi146.stdout: rbd-nbd-2:18.2.1-0.el9.x86_64 2024-05-04T12:06:36.634 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-05-04T12:06:36.634 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-05-04T12:06:36.891 DEBUG:teuthology.parallel:result is None 2024-05-04T12:06:45.709 INFO:teuthology.orchestra.run.smithi079.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 1.2 kB/s | 19 kB 00:15 2024-05-04T12:06:45.734 INFO:teuthology.orchestra.run.smithi079.stdout:(3/13): python3-jsonpatch-1.21-16.el9.noarch.rp 1.6 kB/s | 26 kB 00:15 2024-05-04T12:06:45.759 INFO:teuthology.orchestra.run.smithi079.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 14 kB/s | 222 kB 00:15 2024-05-04T12:06:45.851 INFO:teuthology.orchestra.run.smithi079.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.1 MB/s | 159 kB 00:00 2024-05-04T12:06:45.877 INFO:teuthology.orchestra.run.smithi079.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 459 kB/s | 54 kB 00:00 2024-05-04T12:06:45.902 INFO:teuthology.orchestra.run.smithi079.stdout:(7/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.6 MB/s | 279 kB 00:00 2024-05-04T12:06:46.294 INFO:teuthology.orchestra.run.smithi079.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 36 kB/s | 15 kB 00:00 2024-05-04T12:06:46.361 INFO:teuthology.orchestra.run.smithi079.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 63 kB/s | 32 kB 00:00 2024-05-04T12:06:46.727 INFO:teuthology.orchestra.run.smithi079.stdout:(10/13): python3-google-auth-2.29.0-1.el9.noarc 249 kB/s | 206 kB 00:00 2024-05-04T12:06:46.903 INFO:teuthology.orchestra.run.smithi079.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 109 kB/s | 59 kB 00:00 2024-05-04T12:06:47.345 INFO:teuthology.orchestra.run.smithi079.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 145 kB/s | 90 kB 00:00 2024-05-04T12:06:47.687 INFO:teuthology.orchestra.run.smithi079.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 751 kB/s | 1.0 MB 00:01 2024-05-04T12:06:47.687 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-05-04T12:06:47.688 INFO:teuthology.orchestra.run.smithi079.stdout:Total 127 kB/s | 2.2 MB 00:17 2024-05-04T12:06:47.757 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-05-04T12:06:47.773 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-05-04T12:06:47.774 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-05-04T12:06:47.858 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-05-04T12:06:47.858 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-05-04T12:06:48.435 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-05-04T12:06:48.633 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-05-04T12:06:48.688 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-05-04T12:06:48.742 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-05-04T12:06:48.780 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-05-04T12:06:48.821 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-05-04T12:06:48.924 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-05-04T12:06:49.034 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 7/13 2024-05-04T12:06:49.089 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-05-04T12:06:49.807 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-05-04T12:06:49.854 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-05-04T12:06:49.887 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-05-04T12:06:49.935 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-05-04T12:06:49.957 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-mgr-rook-2:18.2.1-0.el9.noarch 13/13 2024-05-04T12:06:50.601 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.1-0.el9.noarch 13/13 2024-05-04T12:06:50.602 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-mgr-rook-2:18.2.1-0.el9.noarch 1/13 2024-05-04T12:06:50.602 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-05-04T12:06:50.602 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-05-04T12:06:50.602 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-05-04T12:06:50.602 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-05-04T12:06:50.602 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-05-04T12:06:50.602 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-05-04T12:06:50.602 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-05-04T12:06:50.602 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-05-04T12:06:50.602 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 10/13 2024-05-04T12:06:50.602 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-05-04T12:06:50.603 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-05-04T12:06:50.904 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-05-04T12:06:50.904 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-05-04T12:06:50.904 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-05-04T12:06:50.904 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-rook-2:18.2.1-0.el9.noarch 2024-05-04T12:06:50.904 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-05-04T12:06:50.904 INFO:teuthology.orchestra.run.smithi079.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-05-04T12:06:50.905 INFO:teuthology.orchestra.run.smithi079.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-05-04T12:06:50.905 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-05-04T12:06:50.905 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-05-04T12:06:50.905 INFO:teuthology.orchestra.run.smithi079.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-05-04T12:06:50.905 INFO:teuthology.orchestra.run.smithi079.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-05-04T12:06:50.905 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-05-04T12:06:50.905 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-05-04T12:06:50.905 INFO:teuthology.orchestra.run.smithi079.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-05-04T12:06:50.905 INFO:teuthology.orchestra.run.smithi079.stdout: python3-rsa-4.9-2.el9.noarch 2024-05-04T12:06:50.905 INFO:teuthology.orchestra.run.smithi079.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-05-04T12:06:50.905 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-05-04T12:06:50.905 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-05-04T12:06:51.183 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install ceph-mgr-cephadm 2024-05-04T12:06:51.691 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:37 ago on Sat 04 May 2024 12:05:14 PM UTC. 2024-05-04T12:06:51.810 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-05-04T12:06:51.811 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-05-04T12:06:51.811 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repository Size 2024-05-04T12:06:51.811 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-05-04T12:06:51.811 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-05-04T12:06:51.811 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-cephadm noarch 2:18.2.1-0.el9 ceph-noarch 132 k 2024-05-04T12:06:51.811 INFO:teuthology.orchestra.run.smithi079.stdout:Installing dependencies: 2024-05-04T12:06:51.811 INFO:teuthology.orchestra.run.smithi079.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-05-04T12:06:51.812 INFO:teuthology.orchestra.run.smithi079.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-05-04T12:06:51.812 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-05-04T12:06:51.812 INFO:teuthology.orchestra.run.smithi079.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-05-04T12:06:51.812 INFO:teuthology.orchestra.run.smithi079.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-05-04T12:06:51.812 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-05-04T12:06:51.812 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-05-04T12:06:51.812 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-05-04T12:06:51.812 INFO:teuthology.orchestra.run.smithi079.stdout:Install 6 Packages 2024-05-04T12:06:51.812 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-05-04T12:06:51.813 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 7.0 M 2024-05-04T12:06:51.813 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 33 M 2024-05-04T12:06:51.813 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-05-04T12:06:52.221 INFO:teuthology.orchestra.run.smithi079.stdout:(1/6): ceph-mgr-cephadm-18.2.1-0.el9.noarch.rpm 2.0 MB/s | 132 kB 00:00 2024-05-04T12:06:52.755 INFO:teuthology.orchestra.run.smithi079.stdout:(2/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 414 kB/s | 248 kB 00:00 2024-05-04T12:06:53.080 INFO:teuthology.orchestra.run.smithi079.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 177 kB/s | 58 kB 00:00 2024-05-04T12:06:53.138 INFO:teuthology.orchestra.run.smithi079.stdout:(4/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 598 kB/s | 548 kB 00:00 2024-05-04T12:06:53.397 INFO:teuthology.orchestra.run.smithi079.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 161 kB/s | 51 kB 00:00 2024-05-04T12:06:55.364 INFO:teuthology.orchestra.run.smithi079.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 1.9 MB/s | 6.0 MB 00:03 2024-05-04T12:06:55.365 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-05-04T12:06:55.365 INFO:teuthology.orchestra.run.smithi079.stdout:Total 2.0 MB/s | 7.0 MB 00:03 2024-05-04T12:06:55.439 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-05-04T12:06:55.450 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-05-04T12:06:55.450 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-05-04T12:06:55.546 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-05-04T12:06:55.546 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-05-04T12:06:55.846 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-05-04T12:06:56.037 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-05-04T12:06:56.090 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-05-04T12:06:56.603 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-05-04T12:06:56.692 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-05-04T12:06:56.754 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-05-04T12:06:56.777 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-mgr-cephadm-2:18.2.1-0.el9.noarch 6/6 2024-05-04T12:06:57.573 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.1-0.el9.noarch 6/6 2024-05-04T12:06:57.573 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-mgr-cephadm-2:18.2.1-0.el9.noarch 1/6 2024-05-04T12:06:57.574 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-05-04T12:06:57.574 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-05-04T12:06:57.574 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-05-04T12:06:57.574 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-05-04T12:06:57.822 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-05-04T12:06:57.822 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-05-04T12:06:57.822 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-05-04T12:06:57.822 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-cephadm-2:18.2.1-0.el9.noarch 2024-05-04T12:06:57.822 INFO:teuthology.orchestra.run.smithi079.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-05-04T12:06:57.822 INFO:teuthology.orchestra.run.smithi079.stdout: python3-babel-2.9.1-2.el9.noarch 2024-05-04T12:06:57.822 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-05-04T12:06:57.823 INFO:teuthology.orchestra.run.smithi079.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-05-04T12:06:57.823 INFO:teuthology.orchestra.run.smithi079.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-05-04T12:06:57.823 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-05-04T12:06:57.823 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-05-04T12:06:58.096 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install ceph-fuse 2024-05-04T12:06:58.605 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:44 ago on Sat 04 May 2024 12:05:14 PM UTC. 2024-05-04T12:06:58.723 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-05-04T12:06:58.724 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-05-04T12:06:58.725 INFO:teuthology.orchestra.run.smithi079.stdout: Package Architecture Version Repository Size 2024-05-04T12:06:58.725 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-05-04T12:06:58.725 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-05-04T12:06:58.725 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-fuse x86_64 2:18.2.1-0.el9 ceph 839 k 2024-05-04T12:06:58.725 INFO:teuthology.orchestra.run.smithi079.stdout:Installing dependencies: 2024-05-04T12:06:58.725 INFO:teuthology.orchestra.run.smithi079.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-05-04T12:06:58.726 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-05-04T12:06:58.726 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-05-04T12:06:58.726 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-05-04T12:06:58.726 INFO:teuthology.orchestra.run.smithi079.stdout:Install 2 Packages 2024-05-04T12:06:58.726 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-05-04T12:06:58.726 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 920 k 2024-05-04T12:06:58.726 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 2.7 M 2024-05-04T12:06:58.726 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-05-04T12:06:58.865 INFO:teuthology.orchestra.run.smithi079.stdout:(1/2): ceph-fuse-18.2.1-0.el9.x86_64.rpm 11 MB/s | 839 kB 00:00 2024-05-04T12:06:59.216 INFO:teuthology.orchestra.run.smithi079.stdout:(2/2): fuse-2.9.9-15.el9.x86_64.rpm 188 kB/s | 80 kB 00:00 2024-05-04T12:06:59.216 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-05-04T12:06:59.216 INFO:teuthology.orchestra.run.smithi079.stdout:Total 1.8 MB/s | 920 kB 00:00 2024-05-04T12:06:59.228 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-05-04T12:06:59.249 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-05-04T12:06:59.249 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-05-04T12:06:59.325 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-05-04T12:06:59.325 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-05-04T12:06:59.513 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-05-04T12:06:59.638 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-05-04T12:06:59.696 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-fuse-2:18.2.1-0.el9.x86_64 2/2 2024-05-04T12:07:00.322 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-fuse-2:18.2.1-0.el9.x86_64 2/2 2024-05-04T12:07:00.322 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-fuse-2:18.2.1-0.el9.x86_64 1/2 2024-05-04T12:07:00.554 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-05-04T12:07:00.555 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-05-04T12:07:00.555 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-05-04T12:07:00.555 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-fuse-2:18.2.1-0.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-05-04T12:07:00.555 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-05-04T12:07:00.555 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-05-04T12:07:00.815 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install librados-devel 2024-05-04T12:07:01.339 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:47 ago on Sat 04 May 2024 12:05:14 PM UTC. 2024-05-04T12:07:01.462 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-05-04T12:07:01.463 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-05-04T12:07:01.463 INFO:teuthology.orchestra.run.smithi079.stdout: Package Architecture Version Repository Size 2024-05-04T12:07:01.463 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-05-04T12:07:01.463 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-05-04T12:07:01.463 INFO:teuthology.orchestra.run.smithi079.stdout: librados-devel x86_64 2:18.2.1-0.el9 ceph 127 k 2024-05-04T12:07:01.463 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-05-04T12:07:01.463 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-05-04T12:07:01.463 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-05-04T12:07:01.463 INFO:teuthology.orchestra.run.smithi079.stdout:Install 1 Package 2024-05-04T12:07:01.463 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-05-04T12:07:01.464 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 127 k 2024-05-04T12:07:01.464 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 456 k 2024-05-04T12:07:01.464 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-05-04T12:07:01.529 INFO:teuthology.orchestra.run.smithi079.stdout:librados-devel-18.2.1-0.el9.x86_64.rpm 1.9 MB/s | 127 kB 00:00 2024-05-04T12:07:01.529 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-05-04T12:07:01.530 INFO:teuthology.orchestra.run.smithi079.stdout:Total 1.9 MB/s | 127 kB 00:00 2024-05-04T12:07:01.530 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-05-04T12:07:01.535 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-05-04T12:07:01.535 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-05-04T12:07:01.598 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-05-04T12:07:01.598 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-05-04T12:07:01.752 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-05-04T12:07:01.925 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : librados-devel-2:18.2.1-0.el9.x86_64 1/1 2024-05-04T12:07:02.293 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: librados-devel-2:18.2.1-0.el9.x86_64 1/1 2024-05-04T12:07:02.527 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : librados-devel-2:18.2.1-0.el9.x86_64 1/1 2024-05-04T12:07:02.527 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-05-04T12:07:02.527 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-05-04T12:07:02.527 INFO:teuthology.orchestra.run.smithi079.stdout: librados-devel-2:18.2.1-0.el9.x86_64 2024-05-04T12:07:02.527 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-05-04T12:07:02.527 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-05-04T12:07:02.781 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install libcephfs2 2024-05-04T12:07:03.294 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:49 ago on Sat 04 May 2024 12:05:14 PM UTC. 2024-05-04T12:07:03.373 INFO:teuthology.orchestra.run.smithi079.stdout:Package libcephfs2-2:18.2.1-0.el9.x86_64 is already installed. 2024-05-04T12:07:03.414 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-05-04T12:07:03.415 INFO:teuthology.orchestra.run.smithi079.stdout:Nothing to do. 2024-05-04T12:07:03.415 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-05-04T12:07:03.533 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install libcephfs-devel 2024-05-04T12:07:04.048 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:50 ago on Sat 04 May 2024 12:05:14 PM UTC. 2024-05-04T12:07:04.166 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-05-04T12:07:04.167 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-05-04T12:07:04.167 INFO:teuthology.orchestra.run.smithi079.stdout: Package Architecture Version Repository Size 2024-05-04T12:07:04.167 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-05-04T12:07:04.167 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-05-04T12:07:04.167 INFO:teuthology.orchestra.run.smithi079.stdout: libcephfs-devel x86_64 2:18.2.1-0.el9 ceph 31 k 2024-05-04T12:07:04.167 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-05-04T12:07:04.167 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-05-04T12:07:04.167 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-05-04T12:07:04.167 INFO:teuthology.orchestra.run.smithi079.stdout:Install 1 Package 2024-05-04T12:07:04.168 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-05-04T12:07:04.168 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 31 k 2024-05-04T12:07:04.168 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 139 k 2024-05-04T12:07:04.168 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-05-04T12:07:04.241 INFO:teuthology.orchestra.run.smithi079.stdout:libcephfs-devel-18.2.1-0.el9.x86_64.rpm 425 kB/s | 31 kB 00:00 2024-05-04T12:07:04.241 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-05-04T12:07:04.241 INFO:teuthology.orchestra.run.smithi079.stdout:Total 418 kB/s | 31 kB 00:00 2024-05-04T12:07:04.241 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-05-04T12:07:04.245 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-05-04T12:07:04.245 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-05-04T12:07:04.247 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-05-04T12:07:04.247 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-05-04T12:07:04.330 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-05-04T12:07:04.452 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libcephfs-devel-2:18.2.1-0.el9.x86_64 1/1 2024-05-04T12:07:04.762 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: libcephfs-devel-2:18.2.1-0.el9.x86_64 1/1 2024-05-04T12:07:04.967 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libcephfs-devel-2:18.2.1-0.el9.x86_64 1/1 2024-05-04T12:07:04.967 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-05-04T12:07:04.968 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-05-04T12:07:04.968 INFO:teuthology.orchestra.run.smithi079.stdout: libcephfs-devel-2:18.2.1-0.el9.x86_64 2024-05-04T12:07:04.968 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-05-04T12:07:04.968 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-05-04T12:07:05.168 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install librados2 2024-05-04T12:07:05.677 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:51 ago on Sat 04 May 2024 12:05:14 PM UTC. 2024-05-04T12:07:05.755 INFO:teuthology.orchestra.run.smithi079.stdout:Package librados2-2:18.2.1-0.el9.x86_64 is already installed. 2024-05-04T12:07:05.796 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-05-04T12:07:05.797 INFO:teuthology.orchestra.run.smithi079.stdout:Nothing to do. 2024-05-04T12:07:05.797 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-05-04T12:07:05.882 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install librbd1 2024-05-04T12:07:06.392 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:52 ago on Sat 04 May 2024 12:05:14 PM UTC. 2024-05-04T12:07:06.470 INFO:teuthology.orchestra.run.smithi079.stdout:Package librbd1-2:18.2.1-0.el9.x86_64 is already installed. 2024-05-04T12:07:06.511 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-05-04T12:07:06.512 INFO:teuthology.orchestra.run.smithi079.stdout:Nothing to do. 2024-05-04T12:07:06.512 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-05-04T12:07:06.610 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install python3-rados 2024-05-04T12:07:07.123 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:53 ago on Sat 04 May 2024 12:05:14 PM UTC. 2024-05-04T12:07:07.201 INFO:teuthology.orchestra.run.smithi079.stdout:Package python3-rados-2:18.2.1-0.el9.x86_64 is already installed. 2024-05-04T12:07:07.242 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-05-04T12:07:07.243 INFO:teuthology.orchestra.run.smithi079.stdout:Nothing to do. 2024-05-04T12:07:07.243 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-05-04T12:07:07.335 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install python3-rgw 2024-05-04T12:07:07.866 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:53 ago on Sat 04 May 2024 12:05:14 PM UTC. 2024-05-04T12:07:07.944 INFO:teuthology.orchestra.run.smithi079.stdout:Package python3-rgw-2:18.2.1-0.el9.x86_64 is already installed. 2024-05-04T12:07:07.984 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-05-04T12:07:07.985 INFO:teuthology.orchestra.run.smithi079.stdout:Nothing to do. 2024-05-04T12:07:07.985 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-05-04T12:07:08.094 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install python3-cephfs 2024-05-04T12:07:08.605 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:54 ago on Sat 04 May 2024 12:05:14 PM UTC. 2024-05-04T12:07:08.684 INFO:teuthology.orchestra.run.smithi079.stdout:Package python3-cephfs-2:18.2.1-0.el9.x86_64 is already installed. 2024-05-04T12:07:08.724 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-05-04T12:07:08.726 INFO:teuthology.orchestra.run.smithi079.stdout:Nothing to do. 2024-05-04T12:07:08.726 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-05-04T12:07:08.816 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install python3-rbd 2024-05-04T12:07:09.329 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:55 ago on Sat 04 May 2024 12:05:14 PM UTC. 2024-05-04T12:07:09.408 INFO:teuthology.orchestra.run.smithi079.stdout:Package python3-rbd-2:18.2.1-0.el9.x86_64 is already installed. 2024-05-04T12:07:09.449 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-05-04T12:07:09.450 INFO:teuthology.orchestra.run.smithi079.stdout:Nothing to do. 2024-05-04T12:07:09.450 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-05-04T12:07:09.534 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install rbd-fuse 2024-05-04T12:07:10.046 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:56 ago on Sat 04 May 2024 12:05:14 PM UTC. 2024-05-04T12:07:10.165 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-05-04T12:07:10.166 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-05-04T12:07:10.166 INFO:teuthology.orchestra.run.smithi079.stdout: Package Architecture Version Repository Size 2024-05-04T12:07:10.166 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-05-04T12:07:10.166 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-05-04T12:07:10.166 INFO:teuthology.orchestra.run.smithi079.stdout: rbd-fuse x86_64 2:18.2.1-0.el9 ceph 86 k 2024-05-04T12:07:10.166 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-05-04T12:07:10.166 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-05-04T12:07:10.167 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-05-04T12:07:10.167 INFO:teuthology.orchestra.run.smithi079.stdout:Install 1 Package 2024-05-04T12:07:10.167 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-05-04T12:07:10.167 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 86 k 2024-05-04T12:07:10.167 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 226 k 2024-05-04T12:07:10.167 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-05-04T12:07:10.235 INFO:teuthology.orchestra.run.smithi079.stdout:rbd-fuse-18.2.1-0.el9.x86_64.rpm 1.2 MB/s | 86 kB 00:00 2024-05-04T12:07:10.236 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-05-04T12:07:10.236 INFO:teuthology.orchestra.run.smithi079.stdout:Total 1.2 MB/s | 86 kB 00:00 2024-05-04T12:07:10.236 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-05-04T12:07:10.242 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-05-04T12:07:10.243 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-05-04T12:07:10.303 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-05-04T12:07:10.304 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-05-04T12:07:10.442 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-05-04T12:07:10.530 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : rbd-fuse-2:18.2.1-0.el9.x86_64 1/1 2024-05-04T12:07:10.853 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: rbd-fuse-2:18.2.1-0.el9.x86_64 1/1 2024-05-04T12:07:11.095 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : rbd-fuse-2:18.2.1-0.el9.x86_64 1/1 2024-05-04T12:07:11.095 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-05-04T12:07:11.095 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-05-04T12:07:11.095 INFO:teuthology.orchestra.run.smithi079.stdout: rbd-fuse-2:18.2.1-0.el9.x86_64 2024-05-04T12:07:11.095 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-05-04T12:07:11.095 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-05-04T12:07:11.365 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install rbd-mirror 2024-05-04T12:07:11.883 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:57 ago on Sat 04 May 2024 12:05:14 PM UTC. 2024-05-04T12:07:12.004 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-05-04T12:07:12.005 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-05-04T12:07:12.005 INFO:teuthology.orchestra.run.smithi079.stdout: Package Architecture Version Repository Size 2024-05-04T12:07:12.005 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-05-04T12:07:12.005 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-05-04T12:07:12.005 INFO:teuthology.orchestra.run.smithi079.stdout: rbd-mirror x86_64 2:18.2.1-0.el9 ceph 3.0 M 2024-05-04T12:07:12.005 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-05-04T12:07:12.005 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-05-04T12:07:12.005 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-05-04T12:07:12.005 INFO:teuthology.orchestra.run.smithi079.stdout:Install 1 Package 2024-05-04T12:07:12.005 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-05-04T12:07:12.005 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 3.0 M 2024-05-04T12:07:12.005 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 12 M 2024-05-04T12:07:12.005 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-05-04T12:07:12.103 INFO:teuthology.orchestra.run.smithi079.stdout:rbd-mirror-18.2.1-0.el9.x86_64.rpm 31 MB/s | 3.0 MB 00:00 2024-05-04T12:07:12.104 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-05-04T12:07:12.104 INFO:teuthology.orchestra.run.smithi079.stdout:Total 30 MB/s | 3.0 MB 00:00 2024-05-04T12:07:12.104 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-05-04T12:07:12.113 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-05-04T12:07:12.113 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-05-04T12:07:12.193 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-05-04T12:07:12.194 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-05-04T12:07:12.608 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-05-04T12:07:12.772 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : rbd-mirror-2:18.2.1-0.el9.x86_64 1/1 2024-05-04T12:07:12.797 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: rbd-mirror-2:18.2.1-0.el9.x86_64 1/1 2024-05-04T12:07:12.797 INFO:teuthology.orchestra.run.smithi079.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-04T12:07:12.797 INFO:teuthology.orchestra.run.smithi079.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-05-04T12:07:12.797 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-04T12:07:12.797 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-04T12:07:12.797 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-05-04T12:07:13.702 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : rbd-mirror-2:18.2.1-0.el9.x86_64 1/1 2024-05-04T12:07:13.702 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-05-04T12:07:13.702 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-05-04T12:07:13.702 INFO:teuthology.orchestra.run.smithi079.stdout: rbd-mirror-2:18.2.1-0.el9.x86_64 2024-05-04T12:07:13.703 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-05-04T12:07:13.703 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-05-04T12:07:13.977 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install rbd-nbd 2024-05-04T12:07:14.494 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:02:00 ago on Sat 04 May 2024 12:05:14 PM UTC. 2024-05-04T12:07:14.617 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-05-04T12:07:14.618 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-05-04T12:07:14.618 INFO:teuthology.orchestra.run.smithi079.stdout: Package Architecture Version Repository Size 2024-05-04T12:07:14.618 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-05-04T12:07:14.618 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-05-04T12:07:14.619 INFO:teuthology.orchestra.run.smithi079.stdout: rbd-nbd x86_64 2:18.2.1-0.el9 ceph 171 k 2024-05-04T12:07:14.619 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-05-04T12:07:14.619 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-05-04T12:07:14.619 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-05-04T12:07:14.619 INFO:teuthology.orchestra.run.smithi079.stdout:Install 1 Package 2024-05-04T12:07:14.619 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-05-04T12:07:14.619 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 171 k 2024-05-04T12:07:14.619 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 494 k 2024-05-04T12:07:14.619 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-05-04T12:07:14.677 INFO:teuthology.orchestra.run.smithi079.stdout:rbd-nbd-18.2.1-0.el9.x86_64.rpm 2.9 MB/s | 171 kB 00:00 2024-05-04T12:07:14.677 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-05-04T12:07:14.677 INFO:teuthology.orchestra.run.smithi079.stdout:Total 2.8 MB/s | 171 kB 00:00 2024-05-04T12:07:14.678 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-05-04T12:07:14.685 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-05-04T12:07:14.685 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-05-04T12:07:14.747 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-05-04T12:07:14.747 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-05-04T12:07:14.900 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-05-04T12:07:15.002 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : rbd-nbd-2:18.2.1-0.el9.x86_64 1/1 2024-05-04T12:07:15.376 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: rbd-nbd-2:18.2.1-0.el9.x86_64 1/1 2024-05-04T12:07:15.642 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : rbd-nbd-2:18.2.1-0.el9.x86_64 1/1 2024-05-04T12:07:15.642 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-05-04T12:07:15.642 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-05-04T12:07:15.642 INFO:teuthology.orchestra.run.smithi079.stdout: rbd-nbd-2:18.2.1-0.el9.x86_64 2024-05-04T12:07:15.642 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-05-04T12:07:15.642 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-05-04T12:07:15.898 DEBUG:teuthology.parallel:result is None 2024-05-04T12:07:15.899 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-05-04T12:07:15.899 INFO:teuthology.packaging:ref: None 2024-05-04T12:07:15.899 INFO:teuthology.packaging:tag: v18.2.1 2024-05-04T12:07:15.899 INFO:teuthology.packaging:branch: None 2024-05-04T12:07:15.899 INFO:teuthology.packaging:sha1: 09dbd6bba52d19ba471c33bc15c008e6ad158ea6 2024-05-04T12:07:15.900 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=7fe91d5d5842e04be3b4f514d6dd990c54b29c76 2024-05-04T12:07:16.131 DEBUG:teuthology.orchestra.run.smithi079:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-05-04T12:07:16.151 INFO:teuthology.orchestra.run.smithi079.stdout:18.2.1-0.el9 2024-05-04T12:07:16.152 INFO:teuthology.packaging:The installed version of ceph is 18.2.1-0.el9 2024-05-04T12:07:16.152 INFO:teuthology.task.install:The correct ceph version 18.2.1-0 is installed. 2024-05-04T12:07:16.154 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-05-04T12:07:16.154 INFO:teuthology.packaging:ref: None 2024-05-04T12:07:16.154 INFO:teuthology.packaging:tag: v18.2.1 2024-05-04T12:07:16.154 INFO:teuthology.packaging:branch: None 2024-05-04T12:07:16.154 INFO:teuthology.packaging:sha1: 09dbd6bba52d19ba471c33bc15c008e6ad158ea6 2024-05-04T12:07:16.154 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=7fe91d5d5842e04be3b4f514d6dd990c54b29c76 2024-05-04T12:07:16.306 DEBUG:teuthology.orchestra.run.smithi146:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-05-04T12:07:16.331 INFO:teuthology.orchestra.run.smithi146.stdout:18.2.1-0.el9 2024-05-04T12:07:16.331 INFO:teuthology.packaging:The installed version of ceph is 18.2.1-0.el9 2024-05-04T12:07:16.331 INFO:teuthology.task.install:The correct ceph version 18.2.1-0 is installed. 2024-05-04T12:07:16.332 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-05-04T12:07:16.333 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-05-04T12:07:16.333 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-05-04T12:07:16.357 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-05-04T12:07:16.357 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-05-04T12:07:16.398 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-05-04T12:07:16.399 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-05-04T12:07:16.399 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/usr/bin/daemon-helper 2024-05-04T12:07:16.421 DEBUG:teuthology.orchestra.run.smithi079:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-05-04T12:07:16.484 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-05-04T12:07:16.485 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/usr/bin/daemon-helper 2024-05-04T12:07:16.511 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-05-04T12:07:16.577 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-05-04T12:07:16.577 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-05-04T12:07:16.577 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/usr/bin/adjust-ulimits 2024-05-04T12:07:16.602 DEBUG:teuthology.orchestra.run.smithi079:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-05-04T12:07:16.666 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-05-04T12:07:16.666 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/usr/bin/adjust-ulimits 2024-05-04T12:07:16.692 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-05-04T12:07:16.758 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-05-04T12:07:16.759 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-05-04T12:07:16.759 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/usr/bin/stdin-killer 2024-05-04T12:07:16.781 DEBUG:teuthology.orchestra.run.smithi079:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-05-04T12:07:16.845 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-05-04T12:07:16.845 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/usr/bin/stdin-killer 2024-05-04T12:07:16.869 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-05-04T12:07:16.933 INFO:teuthology.run_tasks:Running task print... 2024-05-04T12:07:16.945 INFO:teuthology.task.print:**** done install task... 2024-05-04T12:07:16.945 INFO:teuthology.run_tasks:Running task cephadm... 2024-05-04T12:07:17.040 INFO:tasks.cephadm:Config: {'compiled_cephadm_branch': 'reef', 'conf': {'osd': {'osd_class_default_list': '*', 'osd_class_load_list': '*', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'mon pg warn min per osd': 0}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}}, 'image': 'quay.io/ceph/ceph:v18.2.1', 'roleless': True, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'OSD_DOWN', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN'], 'sha1': '09dbd6bba52d19ba471c33bc15c008e6ad158ea6'} 2024-05-04T12:07:17.040 INFO:tasks.cephadm:Cluster image is quay.io/ceph/ceph:v18.2.1 2024-05-04T12:07:17.041 INFO:tasks.cephadm:Cluster fsid is d9b985c4-0a0e-11ef-bc97-c7b262605968 2024-05-04T12:07:17.041 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-05-04T12:07:17.041 INFO:tasks.cephadm:No mon roles; fabricating mons 2024-05-04T12:07:17.041 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi079': '172.21.15.79', 'mon.smithi146': '172.21.15.146'} 2024-05-04T12:07:17.041 INFO:tasks.cephadm:Normalizing hostnames... 2024-05-04T12:07:17.041 DEBUG:teuthology.orchestra.run.smithi079:> sudo hostname $(hostname -s) 2024-05-04T12:07:17.065 DEBUG:teuthology.orchestra.run.smithi146:> sudo hostname $(hostname -s) 2024-05-04T12:07:17.090 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra for reef 2024-05-04T12:07:17.090 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=09dbd6bba52d19ba471c33bc15c008e6ad158ea6 2024-05-04T12:07:17.312 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '09dbd6bba52d19ba471c33bc15c008e6ad158ea6', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/78943/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-2455-g09dbd6bb', 'node_name': '172.21.2.1+braggi01', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-2455.g09dbd6bb'}, 'url': 'https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2024-04-29-0642/09dbd6bba52d19ba471c33bc15c008e6ad158ea6/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-04-29 15:24:48.681303', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-yuri4-testing-2024-04-29-0642', 'chacra_url': 'https://3.chacra.ceph.com/repos/ceph/wip-yuri4-testing-2024-04-29-0642/09dbd6bba52d19ba471c33bc15c008e6ad158ea6/centos/9/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}] 2024-05-04T12:07:17.623 INFO:tasks.util.chacra:got chacra host 2.chacra.ceph.com, ref reef, sha1 9bc99b09ffc49719013c96c7a9af4027a060bd4b from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&ref=reef 2024-05-04T12:07:17.625 INFO:tasks.cephadm:Discovered cachra url: https://2.chacra.ceph.com/binaries/ceph/reef/9bc99b09ffc49719013c96c7a9af4027a060bd4b/centos/9/x86_64/flavors/default/cephadm 2024-05-04T12:07:17.625 DEBUG:teuthology.orchestra.run.smithi079:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/reef/9bc99b09ffc49719013c96c7a9af4027a060bd4b/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-05-04T12:07:18.016 INFO:teuthology.orchestra.run.smithi079.stdout:-rw-r--r--. 1 ubuntu ubuntu 215335 May 4 12:07 /home/ubuntu/cephtest/cephadm 2024-05-04T12:07:18.016 DEBUG:teuthology.orchestra.run.smithi146:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/reef/9bc99b09ffc49719013c96c7a9af4027a060bd4b/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-05-04T12:07:18.546 INFO:teuthology.orchestra.run.smithi146.stdout:-rw-r--r--. 1 ubuntu ubuntu 215335 May 4 12:07 /home/ubuntu/cephtest/cephadm 2024-05-04T12:07:18.546 DEBUG:teuthology.orchestra.run.smithi079:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-05-04T12:07:18.564 DEBUG:teuthology.orchestra.run.smithi146:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-05-04T12:07:18.592 INFO:tasks.cephadm:Pulling image quay.io/ceph/ceph:v18.2.1 on all hosts... 2024-05-04T12:07:18.592 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 pull 2024-05-04T12:07:18.607 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 pull 2024-05-04T12:07:20.141 INFO:teuthology.orchestra.run.smithi079.stderr:Pulling container image quay.io/ceph/ceph:v18.2.1... 2024-05-04T12:07:20.171 INFO:teuthology.orchestra.run.smithi146.stderr:Pulling container image quay.io/ceph/ceph:v18.2.1... 2024-05-04T12:07:37.804 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-05-04T12:07:37.805 INFO:teuthology.orchestra.run.smithi146.stdout: "ceph_version": "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)", 2024-05-04T12:07:37.805 INFO:teuthology.orchestra.run.smithi146.stdout: "image_id": "5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf", 2024-05-04T12:07:37.805 INFO:teuthology.orchestra.run.smithi146.stdout: "repo_digests": [ 2024-05-04T12:07:37.805 INFO:teuthology.orchestra.run.smithi146.stdout: "quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3", 2024-05-04T12:07:37.805 INFO:teuthology.orchestra.run.smithi146.stdout: "quay.io/ceph/ceph@sha256:e8e55db8b4fd270dbec25bc764437a2a3abb707971c4dba5f559fb83018049dc" 2024-05-04T12:07:37.805 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-05-04T12:07:37.805 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-05-04T12:07:42.197 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-05-04T12:07:42.198 INFO:teuthology.orchestra.run.smithi079.stdout: "ceph_version": "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)", 2024-05-04T12:07:42.198 INFO:teuthology.orchestra.run.smithi079.stdout: "image_id": "5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf", 2024-05-04T12:07:42.198 INFO:teuthology.orchestra.run.smithi079.stdout: "repo_digests": [ 2024-05-04T12:07:42.198 INFO:teuthology.orchestra.run.smithi079.stdout: "quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3", 2024-05-04T12:07:42.198 INFO:teuthology.orchestra.run.smithi079.stdout: "quay.io/ceph/ceph@sha256:e8e55db8b4fd270dbec25bc764437a2a3abb707971c4dba5f559fb83018049dc" 2024-05-04T12:07:42.198 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-05-04T12:07:42.198 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-05-04T12:07:42.215 DEBUG:teuthology.orchestra.run.smithi079:> sudo mkdir -p /etc/ceph 2024-05-04T12:07:42.275 DEBUG:teuthology.orchestra.run.smithi146:> sudo mkdir -p /etc/ceph 2024-05-04T12:07:42.307 DEBUG:teuthology.orchestra.run.smithi079:> sudo chmod 777 /etc/ceph 2024-05-04T12:07:42.331 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod 777 /etc/ceph 2024-05-04T12:07:42.375 INFO:tasks.cephadm:Writing seed config... 2024-05-04T12:07:42.376 INFO:tasks.cephadm: override: [osd] osd_class_default_list = * 2024-05-04T12:07:42.376 INFO:tasks.cephadm: override: [osd] osd_class_load_list = * 2024-05-04T12:07:42.376 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2024-05-04T12:07:42.376 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2024-05-04T12:07:42.376 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2024-05-04T12:07:42.377 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-05-04T12:07:42.377 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-05-04T12:07:42.377 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2024-05-04T12:07:42.377 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2024-05-04T12:07:42.377 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-05-04T12:07:42.377 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-05-04T12:07:42.377 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2024-05-04T12:07:42.377 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-05-04T12:07:42.377 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-05-04T12:07:42.377 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-05-04T12:07:42.377 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-05-04T12:07:42.377 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-05-04T12:07:42.377 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-05-04T12:07:42.377 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-05-04T12:07:42.377 INFO:tasks.cephadm: override: [client] debug client = 20 2024-05-04T12:07:42.378 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-05-04T12:07:42.378 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-05-04T12:07:42.378 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-05-04T12:07:42.378 INFO:tasks.cephadm: override: [global] mon pg warn min per osd = 0 2024-05-04T12:07:42.378 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-05-04T12:07:42.378 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-05-04T12:07:42.378 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-05-04T12:07:42.378 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-05-04T12:07:42.378 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-05-04T12:07:42.378 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-05-04T12:07:42.378 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-05-04T12:07:42.378 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-05-04T12:07:42.378 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-05-04T12:07:42.378 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-05-04T12:07:42.379 INFO:tasks.cephadm: override: [mgr] debug client = 20 2024-05-04T12:07:42.379 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-05-04T12:07:42.379 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-05-04T12:07:42.379 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-05-04T12:07:42.379 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-05-04T12:07:42.379 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-05-04T12:07:42.379 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2024-05-04T12:07:42.379 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-05-04T12:07:42.380 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-05-04T12:07:42.380 DEBUG:teuthology.orchestra.run.smithi079:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-05-04T12:07:42.394 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = d9b985c4-0a0e-11ef-bc97-c7b262605968 mon pg warn min per osd = 0 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true osd_class_default_list = * osd_class_load_list = * bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug client = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-05-04T12:07:42.395 DEBUG:teuthology.orchestra.run.smithi079:mon.smithi079> sudo journalctl -f -n 0 -u ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968@mon.smithi079.service 2024-05-04T12:07:42.437 INFO:tasks.cephadm:Bootstrapping... 2024-05-04T12:07:42.437 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 -v bootstrap --fsid d9b985c4-0a0e-11ef-bc97-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.79 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-05-04T12:07:42.588 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-05-04T12:07:42.588 INFO:teuthology.orchestra.run.smithi079.stdout:cephadm ['--image', 'quay.io/ceph/ceph:v18.2.1', '-v', 'bootstrap', '--fsid', 'd9b985c4-0a0e-11ef-bc97-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-ip', '172.21.15.79', '--skip-admin-label'] 2024-05-04T12:07:42.614 INFO:teuthology.orchestra.run.smithi079.stdout:/bin/podman: stdout 4.9.4-dev 2024-05-04T12:07:42.615 INFO:teuthology.orchestra.run.smithi079.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-05-04T12:07:42.615 INFO:teuthology.orchestra.run.smithi079.stdout:Verifying podman|docker is present... 2024-05-04T12:07:42.639 INFO:teuthology.orchestra.run.smithi079.stdout:/bin/podman: stdout 4.9.4-dev 2024-05-04T12:07:42.639 INFO:teuthology.orchestra.run.smithi079.stdout:Verifying lvm2 is present... 2024-05-04T12:07:42.639 INFO:teuthology.orchestra.run.smithi079.stdout:Verifying time synchronization is in place... 2024-05-04T12:07:42.646 INFO:teuthology.orchestra.run.smithi079.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-05-04T12:07:42.646 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-05-04T12:07:42.652 INFO:teuthology.orchestra.run.smithi079.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-05-04T12:07:42.652 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stdout inactive 2024-05-04T12:07:42.658 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stdout enabled 2024-05-04T12:07:42.664 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stdout active 2024-05-04T12:07:42.664 INFO:teuthology.orchestra.run.smithi079.stdout:Unit chronyd.service is enabled and running 2024-05-04T12:07:42.665 INFO:teuthology.orchestra.run.smithi079.stdout:Repeating the final host check... 2024-05-04T12:07:42.688 INFO:teuthology.orchestra.run.smithi079.stdout:/bin/podman: stdout 4.9.4-dev 2024-05-04T12:07:42.688 INFO:teuthology.orchestra.run.smithi079.stdout:podman (/bin/podman) version 4.9.4 is present 2024-05-04T12:07:42.689 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl is present 2024-05-04T12:07:42.689 INFO:teuthology.orchestra.run.smithi079.stdout:lvcreate is present 2024-05-04T12:07:42.695 INFO:teuthology.orchestra.run.smithi079.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-05-04T12:07:42.695 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-05-04T12:07:42.700 INFO:teuthology.orchestra.run.smithi079.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-05-04T12:07:42.701 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stdout inactive 2024-05-04T12:07:42.706 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stdout enabled 2024-05-04T12:07:42.713 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stdout active 2024-05-04T12:07:42.713 INFO:teuthology.orchestra.run.smithi079.stdout:Unit chronyd.service is enabled and running 2024-05-04T12:07:42.713 INFO:teuthology.orchestra.run.smithi079.stdout:Host looks OK 2024-05-04T12:07:42.713 INFO:teuthology.orchestra.run.smithi079.stdout:Cluster fsid: d9b985c4-0a0e-11ef-bc97-c7b262605968 2024-05-04T12:07:42.713 INFO:teuthology.orchestra.run.smithi079.stdout:Acquiring lock 139715999286272 on /run/cephadm/d9b985c4-0a0e-11ef-bc97-c7b262605968.lock 2024-05-04T12:07:42.713 INFO:teuthology.orchestra.run.smithi079.stdout:Lock 139715999286272 acquired on /run/cephadm/d9b985c4-0a0e-11ef-bc97-c7b262605968.lock 2024-05-04T12:07:42.713 INFO:teuthology.orchestra.run.smithi079.stdout:Verifying IP 172.21.15.79 port 3300 ... 2024-05-04T12:07:42.714 INFO:teuthology.orchestra.run.smithi079.stdout:Verifying IP 172.21.15.79 port 6789 ... 2024-05-04T12:07:42.714 INFO:teuthology.orchestra.run.smithi079.stdout:Base mon IP(s) is [172.21.15.79:3300, 172.21.15.79:6789], mon addrv is [v2:172.21.15.79:3300,v1:172.21.15.79:6789] 2024-05-04T12:07:42.717 INFO:teuthology.orchestra.run.smithi079.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.79 metric 100 2024-05-04T12:07:42.717 INFO:teuthology.orchestra.run.smithi079.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.79 metric 100 2024-05-04T12:07:42.719 INFO:teuthology.orchestra.run.smithi079.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-05-04T12:07:42.719 INFO:teuthology.orchestra.run.smithi079.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 1024 pref medium 2024-05-04T12:07:42.720 INFO:teuthology.orchestra.run.smithi079.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 100 pref medium 2024-05-04T12:07:42.721 INFO:teuthology.orchestra.run.smithi079.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-05-04T12:07:42.721 INFO:teuthology.orchestra.run.smithi079.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-05-04T12:07:42.721 INFO:teuthology.orchestra.run.smithi079.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-05-04T12:07:42.722 INFO:teuthology.orchestra.run.smithi079.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-05-04T12:07:42.722 INFO:teuthology.orchestra.run.smithi079.stdout:/sbin/ip: stdout inet6 fe80::d9a9:5e05:f280:83dd/64 scope link noprefixroute 2024-05-04T12:07:42.722 INFO:teuthology.orchestra.run.smithi079.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-05-04T12:07:42.722 INFO:teuthology.orchestra.run.smithi079.stdout:Mon IP `172.21.15.79` is in CIDR network `172.21.0.0/20` 2024-05-04T12:07:42.722 INFO:teuthology.orchestra.run.smithi079.stdout:Mon IP `172.21.15.79` is in CIDR network `172.21.0.0/20` 2024-05-04T12:07:42.722 INFO:teuthology.orchestra.run.smithi079.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-05-04T12:07:42.723 INFO:teuthology.orchestra.run.smithi079.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-05-04T12:07:42.724 INFO:teuthology.orchestra.run.smithi079.stdout:Pulling container image quay.io/ceph/ceph:v18.2.1... 2024-05-04T12:07:43.559 INFO:teuthology.orchestra.run.smithi079.stdout:/bin/podman: stdout 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf 2024-05-04T12:07:43.560 INFO:teuthology.orchestra.run.smithi079.stdout:/bin/podman: stderr Trying to pull quay.io/ceph/ceph:v18.2.1... 2024-05-04T12:07:43.560 INFO:teuthology.orchestra.run.smithi079.stdout:/bin/podman: stderr Getting image source signatures 2024-05-04T12:07:43.560 INFO:teuthology.orchestra.run.smithi079.stdout:/bin/podman: stderr Copying blob sha256:a733d3c618b71f19c168ebecd1953429dce2c1631835ca182e9551c36dce5989 2024-05-04T12:07:43.560 INFO:teuthology.orchestra.run.smithi079.stdout:/bin/podman: stderr Copying blob sha256:7feca07754707458c3945cf0062cf4dabc512f6d90fe1a9a1370b362b6011124 2024-05-04T12:07:43.560 INFO:teuthology.orchestra.run.smithi079.stdout:/bin/podman: stderr Copying config sha256:5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf 2024-05-04T12:07:43.560 INFO:teuthology.orchestra.run.smithi079.stdout:/bin/podman: stderr Writing manifest to image destination 2024-05-04T12:07:44.482 INFO:teuthology.orchestra.run.smithi079.stdout:ceph: stdout ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable) 2024-05-04T12:07:44.482 INFO:teuthology.orchestra.run.smithi079.stdout:Ceph version: ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable) 2024-05-04T12:07:44.482 INFO:teuthology.orchestra.run.smithi079.stdout:Extracting ceph user uid/gid from container image... 2024-05-04T12:07:45.371 INFO:teuthology.orchestra.run.smithi079.stdout:stat: stdout 167 167 2024-05-04T12:07:45.372 INFO:teuthology.orchestra.run.smithi079.stdout:Creating initial keys... 2024-05-04T12:07:46.286 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph-authtool: stdout AQARJTZm+5+eKhAAynfO636ujbT9ki/lkwm9sA== 2024-05-04T12:07:47.210 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph-authtool: stdout AQASJTZmLIEpJRAAMfbcFXmKY0wiEnHwQcQTYw== 2024-05-04T12:07:48.100 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph-authtool: stdout AQATJTZmbLAOHhAA4tRhWJHYYNKa65qdEvXoPQ== 2024-05-04T12:07:48.101 INFO:teuthology.orchestra.run.smithi079.stdout:Creating initial monmap... 2024-05-04T12:07:48.982 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-05-04T12:07:48.982 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = pacific 2024-05-04T12:07:48.982 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to d9b985c4-0a0e-11ef-bc97-c7b262605968 2024-05-04T12:07:48.982 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-05-04T12:07:48.983 INFO:teuthology.orchestra.run.smithi079.stdout:monmaptool for smithi079 [v2:172.21.15.79:3300,v1:172.21.15.79:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-05-04T12:07:48.983 INFO:teuthology.orchestra.run.smithi079.stdout:setting min_mon_release = pacific 2024-05-04T12:07:48.983 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/monmaptool: set fsid to d9b985c4-0a0e-11ef-bc97-c7b262605968 2024-05-04T12:07:48.983 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-05-04T12:07:48.983 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-05-04T12:07:48.983 INFO:teuthology.orchestra.run.smithi079.stdout:Creating mon... 2024-05-04T12:07:49.864 INFO:teuthology.orchestra.run.smithi079.stdout:create mon.smithi079 on 2024-05-04T12:07:50.067 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-05-04T12:07:50.222 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-05-04T12:07:50.390 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968.target → /etc/systemd/system/ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968.target. 2024-05-04T12:07:50.391 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968.target → /etc/systemd/system/ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968.target. 2024-05-04T12:07:50.644 INFO:teuthology.orchestra.run.smithi079.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968@mon.smithi079 2024-05-04T12:07:50.645 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stderr Failed to reset failed state of unit ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968@mon.smithi079.service: Unit ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968@mon.smithi079.service not loaded. 2024-05-04T12:07:50.811 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968.target.wants/ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968@mon.smithi079.service → /etc/systemd/system/ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968@.service. 2024-05-04T12:07:51.316 INFO:teuthology.orchestra.run.smithi079.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-05-04T12:07:51.316 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stdout disabled 2024-05-04T12:07:51.324 INFO:teuthology.orchestra.run.smithi079.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-05-04T12:07:51.324 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stdout inactive 2024-05-04T12:07:51.324 INFO:teuthology.orchestra.run.smithi079.stdout:firewalld.service is not enabled 2024-05-04T12:07:51.324 INFO:teuthology.orchestra.run.smithi079.stdout:Not possible to enable service . firewalld.service is not available 2024-05-04T12:07:51.325 INFO:teuthology.orchestra.run.smithi079.stdout:Waiting for mon to start... 2024-05-04T12:07:51.325 INFO:teuthology.orchestra.run.smithi079.stdout:Waiting for mon... 2024-05-04T12:07:51.426 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:51 smithi079 systemd[1]: Started Ceph mon.smithi079 for d9b985c4-0a0e-11ef-bc97-c7b262605968. 2024-05-04T12:07:52.383 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout cluster: 2024-05-04T12:07:52.383 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout id: d9b985c4-0a0e-11ef-bc97-c7b262605968 2024-05-04T12:07:52.383 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-05-04T12:07:52.383 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout 2024-05-04T12:07:52.383 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout services: 2024-05-04T12:07:52.383 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum smithi079 (age 0.545393s) 2024-05-04T12:07:52.384 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-05-04T12:07:52.384 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-05-04T12:07:52.384 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout 2024-05-04T12:07:52.384 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout data: 2024-05-04T12:07:52.384 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-05-04T12:07:52.384 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-05-04T12:07:52.384 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-05-04T12:07:52.384 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout pgs: 2024-05-04T12:07:52.384 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout 2024-05-04T12:07:52.384 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:51.752+0000 7fad82e61700 1 Processor -- start 2024-05-04T12:07:52.384 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:51.752+0000 7fad82e61700 1 -- start start 2024-05-04T12:07:52.384 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:51.752+0000 7fad82e61700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fad7c105470 0x7fad7c105890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:07:52.384 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:51.752+0000 7fad82e61700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad7c105e60 con 0x7fad7c105470 2024-05-04T12:07:52.385 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:51.753+0000 7fad80bfd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fad7c105470 0x7fad7c105890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:07:52.385 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:51.753+0000 7fad80bfd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fad7c105470 0x7fad7c105890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:35450/0 (socket says 172.21.15.79:35450) 2024-05-04T12:07:52.385 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:51.753+0000 7fad80bfd700 1 -- 172.21.15.79:0/1426037663 learned_addr learned my addr 172.21.15.79:0/1426037663 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:07:52.385 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:51.753+0000 7fad80bfd700 1 -- 172.21.15.79:0/1426037663 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad7c106670 con 0x7fad7c105470 2024-05-04T12:07:52.385 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:51.753+0000 7fad80bfd700 1 --2- 172.21.15.79:0/1426037663 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fad7c105470 0x7fad7c105890 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fad700098a0 tx=0x7fad70009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6eac0eff096a8c7a server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:07:52.385 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:51.753+0000 7fad7b7fe700 1 -- 172.21.15.79:0/1426037663 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fad70004180 con 0x7fad7c105470 2024-05-04T12:07:52.385 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:51.753+0000 7fad7b7fe700 1 -- 172.21.15.79:0/1426037663 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad700042e0 con 0x7fad7c105470 2024-05-04T12:07:52.385 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:51.753+0000 7fad7b7fe700 1 -- 172.21.15.79:0/1426037663 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fad700036a0 con 0x7fad7c105470 2024-05-04T12:07:52.385 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:51.754+0000 7fad82e61700 1 -- 172.21.15.79:0/1426037663 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fad7c105470 msgr2=0x7fad7c105890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:07:52.385 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:51.754+0000 7fad82e61700 1 --2- 172.21.15.79:0/1426037663 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fad7c105470 0x7fad7c105890 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fad700098a0 tx=0x7fad70009bb0 comp rx=0 tx=0).stop 2024-05-04T12:07:52.385 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:51.754+0000 7fad82e61700 1 -- 172.21.15.79:0/1426037663 shutdown_connections 2024-05-04T12:07:52.385 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:51.754+0000 7fad82e61700 1 --2- 172.21.15.79:0/1426037663 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fad7c105470 0x7fad7c105890 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:07:52.385 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:51.754+0000 7fad82e61700 1 -- 172.21.15.79:0/1426037663 >> 172.21.15.79:0/1426037663 conn(0x7fad7c1009f0 msgr2=0x7fad7c102e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:07:52.386 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:51.754+0000 7fad82e61700 1 -- 172.21.15.79:0/1426037663 shutdown_connections 2024-05-04T12:07:52.386 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:51.754+0000 7fad82e61700 1 -- 172.21.15.79:0/1426037663 wait complete. 2024-05-04T12:07:52.386 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:51.755+0000 7fad82e61700 1 Processor -- start 2024-05-04T12:07:52.386 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:51.755+0000 7fad82e61700 1 -- start start 2024-05-04T12:07:52.386 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:51.755+0000 7fad82e61700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fad7c105470 0x7fad7c18e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:07:52.386 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:51.755+0000 7fad82e61700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad7c18e650 con 0x7fad7c105470 2024-05-04T12:07:52.386 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:51.755+0000 7fad80bfd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fad7c105470 0x7fad7c18e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:07:52.386 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:51.755+0000 7fad80bfd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fad7c105470 0x7fad7c18e110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:35464/0 (socket says 172.21.15.79:35464) 2024-05-04T12:07:52.386 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:51.755+0000 7fad80bfd700 1 -- 172.21.15.79:0/3077011856 learned_addr learned my addr 172.21.15.79:0/3077011856 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:07:52.386 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:51.755+0000 7fad80bfd700 1 -- 172.21.15.79:0/3077011856 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad70009550 con 0x7fad7c105470 2024-05-04T12:07:52.386 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:51.755+0000 7fad80bfd700 1 --2- 172.21.15.79:0/3077011856 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fad7c105470 0x7fad7c18e110 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fad70004000 tx=0x7fad70004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:07:52.386 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:51.756+0000 7fad79ffb700 1 -- 172.21.15.79:0/3077011856 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fad70003ce0 con 0x7fad7c105470 2024-05-04T12:07:52.386 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:51.756+0000 7fad79ffb700 1 -- 172.21.15.79:0/3077011856 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad70020070 con 0x7fad7c105470 2024-05-04T12:07:52.386 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:51.756+0000 7fad82e61700 1 -- 172.21.15.79:0/3077011856 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad7c18c760 con 0x7fad7c105470 2024-05-04T12:07:52.387 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:51.756+0000 7fad79ffb700 1 -- 172.21.15.79:0/3077011856 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fad70011630 con 0x7fad7c105470 2024-05-04T12:07:52.387 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:51.756+0000 7fad82e61700 1 -- 172.21.15.79:0/3077011856 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad7c18cc00 con 0x7fad7c105470 2024-05-04T12:07:52.387 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:51.756+0000 7fad79ffb700 1 -- 172.21.15.79:0/3077011856 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fad70022030 con 0x7fad7c105470 2024-05-04T12:07:52.387 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:51.756+0000 7fad79ffb700 1 -- 172.21.15.79:0/3077011856 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fad70025d40 con 0x7fad7c105470 2024-05-04T12:07:52.387 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:51.757+0000 7fad82e61700 1 -- 172.21.15.79:0/3077011856 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad7c0623c0 con 0x7fad7c105470 2024-05-04T12:07:52.387 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:51.759+0000 7fad79ffb700 1 -- 172.21.15.79:0/3077011856 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7fad70025480 con 0x7fad7c105470 2024-05-04T12:07:52.387 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:51.828+0000 7fad82e61700 1 -- 172.21.15.79:0/3077011856 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7fad7c192f60 con 0x7fad7c105470 2024-05-04T12:07:52.387 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:51.828+0000 7fad79ffb700 1 -- 172.21.15.79:0/3077011856 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+325 (secure 0 0 0) 0x7fad70011bd0 con 0x7fad7c105470 2024-05-04T12:07:52.387 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:51.829+0000 7fad82e61700 1 -- 172.21.15.79:0/3077011856 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fad7c105470 msgr2=0x7fad7c18e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:07:52.387 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:51.829+0000 7fad82e61700 1 --2- 172.21.15.79:0/3077011856 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fad7c105470 0x7fad7c18e110 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fad70004000 tx=0x7fad70004750 comp rx=0 tx=0).stop 2024-05-04T12:07:52.387 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:51.829+0000 7fad82e61700 1 -- 172.21.15.79:0/3077011856 shutdown_connections 2024-05-04T12:07:52.387 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:51.829+0000 7fad82e61700 1 --2- 172.21.15.79:0/3077011856 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fad7c105470 0x7fad7c18e110 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:07:52.387 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:51.829+0000 7fad82e61700 1 -- 172.21.15.79:0/3077011856 >> 172.21.15.79:0/3077011856 conn(0x7fad7c1009f0 msgr2=0x7fad7c074030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:07:52.387 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:51.829+0000 7fad82e61700 1 -- 172.21.15.79:0/3077011856 shutdown_connections 2024-05-04T12:07:52.388 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:51.829+0000 7fad82e61700 1 -- 172.21.15.79:0/3077011856 wait complete. 2024-05-04T12:07:52.388 INFO:teuthology.orchestra.run.smithi079.stdout:mon is available 2024-05-04T12:07:52.388 INFO:teuthology.orchestra.run.smithi079.stdout:Assimilating anything we can from ceph.conf... 2024-05-04T12:07:52.492 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:52 smithi079 ceph-mon[26849]: mon.smithi079 is new leader, mons smithi079 in quorum (ranks 0) 2024-05-04T12:07:52.492 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:52 smithi079 ceph-mon[26849]: monmap e1: 1 mons at {smithi079=[v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-05-04T12:07:52.492 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:52 smithi079 ceph-mon[26849]: fsmap 2024-05-04T12:07:52.492 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:52 smithi079 ceph-mon[26849]: osdmap e1: 0 total, 0 up, 0 in 2024-05-04T12:07:52.492 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:52 smithi079 ceph-mon[26849]: mgrmap e1: no daemons active 2024-05-04T12:07:52.492 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:52 smithi079 ceph-mon[26849]: from='client.? 172.21.15.79:0/3077011856' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-05-04T12:07:53.400 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout 2024-05-04T12:07:53.401 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout [global] 2024-05-04T12:07:53.401 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout fsid = d9b985c4-0a0e-11ef-bc97-c7b262605968 2024-05-04T12:07:53.401 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.79:3300,v1:172.21.15.79:6789] 2024-05-04T12:07:53.401 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-05-04T12:07:53.401 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-05-04T12:07:53.401 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-05-04T12:07:53.401 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-05-04T12:07:53.401 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout 2024-05-04T12:07:53.401 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout [mgr] 2024-05-04T12:07:53.401 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-05-04T12:07:53.401 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout 2024-05-04T12:07:53.401 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout [osd] 2024-05-04T12:07:53.401 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-05-04T12:07:53.401 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-05-04T12:07:53.402 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-05-04T12:07:53.402 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-05-04T12:07:53.402 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-05-04T12:07:53.402 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:52.776+0000 7feaebf5d700 1 Processor -- start 2024-05-04T12:07:53.402 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:52.776+0000 7feaebf5d700 1 -- start start 2024-05-04T12:07:53.402 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:52.776+0000 7feaebf5d700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7feae4105460 0x7feae4105880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:07:53.402 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:52.776+0000 7feaebf5d700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feae4105e50 con 0x7feae4105460 2024-05-04T12:07:53.402 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:52.776+0000 7feae9cf9700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7feae4105460 0x7feae4105880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:07:53.402 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:52.776+0000 7feae9cf9700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7feae4105460 0x7feae4105880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:35478/0 (socket says 172.21.15.79:35478) 2024-05-04T12:07:53.402 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:52.776+0000 7feae9cf9700 1 -- 172.21.15.79:0/3416861132 learned_addr learned my addr 172.21.15.79:0/3416861132 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:07:53.402 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:52.777+0000 7feae9cf9700 1 -- 172.21.15.79:0/3416861132 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feae4106660 con 0x7feae4105460 2024-05-04T12:07:53.402 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:52.777+0000 7feae9cf9700 1 --2- 172.21.15.79:0/3416861132 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7feae4105460 0x7feae4105880 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7feae00098a0 tx=0x7feae0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae70c998a3c60fb4 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:07:53.402 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:52.777+0000 7feae8cf7700 1 -- 172.21.15.79:0/3416861132 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feae0004180 con 0x7feae4105460 2024-05-04T12:07:53.403 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:52.777+0000 7feae8cf7700 1 -- 172.21.15.79:0/3416861132 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feae00042e0 con 0x7feae4105460 2024-05-04T12:07:53.403 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:52.777+0000 7feae8cf7700 1 -- 172.21.15.79:0/3416861132 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feae00036a0 con 0x7feae4105460 2024-05-04T12:07:53.403 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:52.777+0000 7feaebf5d700 1 -- 172.21.15.79:0/3416861132 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7feae4105460 msgr2=0x7feae4105880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:07:53.403 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:52.777+0000 7feaebf5d700 1 --2- 172.21.15.79:0/3416861132 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7feae4105460 0x7feae4105880 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7feae00098a0 tx=0x7feae0009bb0 comp rx=0 tx=0).stop 2024-05-04T12:07:53.403 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:52.777+0000 7feaebf5d700 1 -- 172.21.15.79:0/3416861132 shutdown_connections 2024-05-04T12:07:53.403 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:52.777+0000 7feaebf5d700 1 --2- 172.21.15.79:0/3416861132 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7feae4105460 0x7feae4105880 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:07:53.403 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:52.777+0000 7feaebf5d700 1 -- 172.21.15.79:0/3416861132 >> 172.21.15.79:0/3416861132 conn(0x7feae4100a20 msgr2=0x7feae4102e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:07:53.403 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:52.778+0000 7feaebf5d700 1 -- 172.21.15.79:0/3416861132 shutdown_connections 2024-05-04T12:07:53.403 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:52.778+0000 7feaebf5d700 1 -- 172.21.15.79:0/3416861132 wait complete. 2024-05-04T12:07:53.404 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:52.778+0000 7feaebf5d700 1 Processor -- start 2024-05-04T12:07:53.404 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:52.778+0000 7feaebf5d700 1 -- start start 2024-05-04T12:07:53.404 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:52.778+0000 7feaebf5d700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7feae4105460 0x7feae4199980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:07:53.404 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:52.778+0000 7feaebf5d700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feae4199ec0 con 0x7feae4105460 2024-05-04T12:07:53.404 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:52.778+0000 7feae9cf9700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7feae4105460 0x7feae4199980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:07:53.404 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:52.778+0000 7feae9cf9700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7feae4105460 0x7feae4199980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:35490/0 (socket says 172.21.15.79:35490) 2024-05-04T12:07:53.404 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:52.778+0000 7feae9cf9700 1 -- 172.21.15.79:0/2066032705 learned_addr learned my addr 172.21.15.79:0/2066032705 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:07:53.404 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:52.779+0000 7feae9cf9700 1 -- 172.21.15.79:0/2066032705 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feae0009550 con 0x7feae4105460 2024-05-04T12:07:53.404 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:52.779+0000 7feae9cf9700 1 --2- 172.21.15.79:0/2066032705 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7feae4105460 0x7feae4199980 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7feae0004000 tx=0x7feae0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:07:53.404 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:52.779+0000 7feadaffd700 1 -- 172.21.15.79:0/2066032705 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feae0003ce0 con 0x7feae4105460 2024-05-04T12:07:53.404 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:52.779+0000 7feadaffd700 1 -- 172.21.15.79:0/2066032705 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feae0020070 con 0x7feae4105460 2024-05-04T12:07:53.405 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:52.779+0000 7feadaffd700 1 -- 172.21.15.79:0/2066032705 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feae0011630 con 0x7feae4105460 2024-05-04T12:07:53.405 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:52.779+0000 7feaebf5d700 1 -- 172.21.15.79:0/2066032705 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feae419a0c0 con 0x7feae4105460 2024-05-04T12:07:53.405 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:52.779+0000 7feaebf5d700 1 -- 172.21.15.79:0/2066032705 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feae419a560 con 0x7feae4105460 2024-05-04T12:07:53.405 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:52.779+0000 7feadaffd700 1 -- 172.21.15.79:0/2066032705 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7feae0022030 con 0x7feae4105460 2024-05-04T12:07:53.405 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:52.779+0000 7feadaffd700 1 -- 172.21.15.79:0/2066032705 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7feae0025d40 con 0x7feae4105460 2024-05-04T12:07:53.405 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:52.780+0000 7feaebf5d700 1 -- 172.21.15.79:0/2066032705 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feae4193200 con 0x7feae4105460 2024-05-04T12:07:53.405 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:52.782+0000 7feadaffd700 1 -- 172.21.15.79:0/2066032705 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7feae0025480 con 0x7feae4105460 2024-05-04T12:07:53.405 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:52.848+0000 7feaebf5d700 1 -- 172.21.15.79:0/2066032705 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7feae40623c0 con 0x7feae4105460 2024-05-04T12:07:53.405 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:52.852+0000 7feadaffd700 1 -- 172.21.15.79:0/2066032705 <== mon.0 v2:172.21.15.79:3300/0 7 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7feae001b450 con 0x7feae4105460 2024-05-04T12:07:53.405 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:52.852+0000 7feadaffd700 1 -- 172.21.15.79:0/2066032705 <== mon.0 v2:172.21.15.79:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+429 (secure 0 0 0) 0x7feae0036020 con 0x7feae4105460 2024-05-04T12:07:53.405 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:52.853+0000 7feaebf5d700 1 -- 172.21.15.79:0/2066032705 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7feae4105460 msgr2=0x7feae4199980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:07:53.405 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:52.853+0000 7feaebf5d700 1 --2- 172.21.15.79:0/2066032705 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7feae4105460 0x7feae4199980 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7feae0004000 tx=0x7feae0004750 comp rx=0 tx=0).stop 2024-05-04T12:07:53.405 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:52.853+0000 7feaebf5d700 1 -- 172.21.15.79:0/2066032705 shutdown_connections 2024-05-04T12:07:53.405 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:52.853+0000 7feaebf5d700 1 --2- 172.21.15.79:0/2066032705 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7feae4105460 0x7feae4199980 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:07:53.406 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:52.853+0000 7feaebf5d700 1 -- 172.21.15.79:0/2066032705 >> 172.21.15.79:0/2066032705 conn(0x7feae4100a20 msgr2=0x7feae4079840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:07:53.406 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:52.853+0000 7feaebf5d700 1 -- 172.21.15.79:0/2066032705 shutdown_connections 2024-05-04T12:07:53.406 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:52.853+0000 7feaebf5d700 1 -- 172.21.15.79:0/2066032705 wait complete. 2024-05-04T12:07:53.406 INFO:teuthology.orchestra.run.smithi079.stdout:Generating new minimal ceph.conf... 2024-05-04T12:07:53.492 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:53 smithi079 ceph-mon[26849]: from='client.? 172.21.15.79:0/2066032705' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-05-04T12:07:53.492 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:53 smithi079 ceph-mon[26849]: from='client.? 172.21.15.79:0/2066032705' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-05-04T12:07:54.364 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:53.766+0000 7f6139ed7700 1 Processor -- start 2024-05-04T12:07:54.364 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:53.766+0000 7f6139ed7700 1 -- start start 2024-05-04T12:07:54.364 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:53.766+0000 7f6139ed7700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f613407bea0 0x7f613407a300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:07:54.364 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:53.766+0000 7f6139ed7700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f613407a8d0 con 0x7f613407bea0 2024-05-04T12:07:54.364 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:53.766+0000 7f61337fe700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f613407bea0 0x7f613407a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:07:54.364 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:53.767+0000 7f61337fe700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f613407bea0 0x7f613407a300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:44858/0 (socket says 172.21.15.79:44858) 2024-05-04T12:07:54.364 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:53.767+0000 7f61337fe700 1 -- 172.21.15.79:0/3478338366 learned_addr learned my addr 172.21.15.79:0/3478338366 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:07:54.364 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:53.767+0000 7f61337fe700 1 -- 172.21.15.79:0/3478338366 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f613407aa10 con 0x7f613407bea0 2024-05-04T12:07:54.364 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:53.767+0000 7f61337fe700 1 --2- 172.21.15.79:0/3478338366 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f613407bea0 0x7f613407a300 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f6124009e80 tx=0x7f612400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14906a20d80de749 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:07:54.365 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:53.767+0000 7f61327fc700 1 -- 172.21.15.79:0/3478338366 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f612400e070 con 0x7f613407bea0 2024-05-04T12:07:54.365 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:53.767+0000 7f61327fc700 1 -- 172.21.15.79:0/3478338366 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f6124003e90 con 0x7f613407bea0 2024-05-04T12:07:54.365 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:53.767+0000 7f61327fc700 1 -- 172.21.15.79:0/3478338366 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6124004380 con 0x7f613407bea0 2024-05-04T12:07:54.365 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:53.768+0000 7f6139ed7700 1 -- 172.21.15.79:0/3478338366 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f613407bea0 msgr2=0x7f613407a300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:07:54.365 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:53.768+0000 7f6139ed7700 1 --2- 172.21.15.79:0/3478338366 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f613407bea0 0x7f613407a300 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f6124009e80 tx=0x7f612400b330 comp rx=0 tx=0).stop 2024-05-04T12:07:54.365 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:53.768+0000 7f6139ed7700 1 -- 172.21.15.79:0/3478338366 shutdown_connections 2024-05-04T12:07:54.365 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:53.768+0000 7f6139ed7700 1 --2- 172.21.15.79:0/3478338366 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f613407bea0 0x7f613407a300 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:07:54.365 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:53.768+0000 7f6139ed7700 1 -- 172.21.15.79:0/3478338366 >> 172.21.15.79:0/3478338366 conn(0x7f6134101b40 msgr2=0x7f6134103f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:07:54.365 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:53.768+0000 7f6139ed7700 1 -- 172.21.15.79:0/3478338366 shutdown_connections 2024-05-04T12:07:54.365 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:53.768+0000 7f6139ed7700 1 -- 172.21.15.79:0/3478338366 wait complete. 2024-05-04T12:07:54.365 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:53.768+0000 7f6139ed7700 1 Processor -- start 2024-05-04T12:07:54.365 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:53.768+0000 7f6139ed7700 1 -- start start 2024-05-04T12:07:54.365 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:53.769+0000 7f6139ed7700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f613407bea0 0x7f613419c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:07:54.365 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:53.769+0000 7f6139ed7700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f613419c920 con 0x7f613407bea0 2024-05-04T12:07:54.366 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:53.769+0000 7f61337fe700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f613407bea0 0x7f613419c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:07:54.366 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:53.769+0000 7f61337fe700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f613407bea0 0x7f613419c3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:44860/0 (socket says 172.21.15.79:44860) 2024-05-04T12:07:54.366 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:53.769+0000 7f61337fe700 1 -- 172.21.15.79:0/1026410092 learned_addr learned my addr 172.21.15.79:0/1026410092 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:07:54.366 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:53.769+0000 7f61337fe700 1 -- 172.21.15.79:0/1026410092 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6124009b30 con 0x7f613407bea0 2024-05-04T12:07:54.366 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:53.769+0000 7f61337fe700 1 --2- 172.21.15.79:0/1026410092 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f613407bea0 0x7f613419c3e0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f6124009e50 tx=0x7f6124004920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:07:54.366 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:53.769+0000 7f6130ff9700 1 -- 172.21.15.79:0/1026410092 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f612400e040 con 0x7f613407bea0 2024-05-04T12:07:54.366 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:53.769+0000 7f6139ed7700 1 -- 172.21.15.79:0/1026410092 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f613419cb20 con 0x7f613407bea0 2024-05-04T12:07:54.366 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:53.769+0000 7f6130ff9700 1 -- 172.21.15.79:0/1026410092 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f6124003cf0 con 0x7f613407bea0 2024-05-04T12:07:54.366 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:53.769+0000 7f6130ff9700 1 -- 172.21.15.79:0/1026410092 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f612401b770 con 0x7f613407bea0 2024-05-04T12:07:54.366 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:53.769+0000 7f6139ed7700 1 -- 172.21.15.79:0/1026410092 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f613419d020 con 0x7f613407bea0 2024-05-04T12:07:54.366 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:53.769+0000 7f6130ff9700 1 -- 172.21.15.79:0/1026410092 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f6124019040 con 0x7f613407bea0 2024-05-04T12:07:54.366 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:53.769+0000 7f6130ff9700 1 -- 172.21.15.79:0/1026410092 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f612401a6d0 con 0x7f613407bea0 2024-05-04T12:07:54.366 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:53.770+0000 7f6139ed7700 1 -- 172.21.15.79:0/1026410092 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f613410c880 con 0x7f613407bea0 2024-05-04T12:07:54.367 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:53.772+0000 7f6130ff9700 1 -- 172.21.15.79:0/1026410092 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7f6124029060 con 0x7f613407bea0 2024-05-04T12:07:54.367 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:53.840+0000 7f6139ed7700 1 -- 172.21.15.79:0/1026410092 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f613410ca70 con 0x7f613407bea0 2024-05-04T12:07:54.367 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:53.840+0000 7f6130ff9700 1 -- 172.21.15.79:0/1026410092 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+175 (secure 0 0 0) 0x7f612401f070 con 0x7f613407bea0 2024-05-04T12:07:54.367 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:53.841+0000 7f6139ed7700 1 -- 172.21.15.79:0/1026410092 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f613407bea0 msgr2=0x7f613419c3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:07:54.367 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:53.841+0000 7f6139ed7700 1 --2- 172.21.15.79:0/1026410092 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f613407bea0 0x7f613419c3e0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f6124009e50 tx=0x7f6124004920 comp rx=0 tx=0).stop 2024-05-04T12:07:54.367 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:53.841+0000 7f6139ed7700 1 -- 172.21.15.79:0/1026410092 shutdown_connections 2024-05-04T12:07:54.367 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:53.841+0000 7f6139ed7700 1 --2- 172.21.15.79:0/1026410092 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f613407bea0 0x7f613419c3e0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:07:54.367 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:53.841+0000 7f6139ed7700 1 -- 172.21.15.79:0/1026410092 >> 172.21.15.79:0/1026410092 conn(0x7f6134101b40 msgr2=0x7f61341028f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:07:54.367 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:53.841+0000 7f6139ed7700 1 -- 172.21.15.79:0/1026410092 shutdown_connections 2024-05-04T12:07:54.367 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:53.841+0000 7f6139ed7700 1 -- 172.21.15.79:0/1026410092 wait complete. 2024-05-04T12:07:54.367 INFO:teuthology.orchestra.run.smithi079.stdout:Restarting the monitor... 2024-05-04T12:07:54.627 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:54 smithi079 ceph-mon[26849]: from='client.? 172.21.15.79:0/1026410092' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-04T12:07:54.627 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:54 smithi079 systemd[1]: Stopping Ceph mon.smithi079 for d9b985c4-0a0e-11ef-bc97-c7b262605968... 2024-05-04T12:07:54.945 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[26845]: 2024-05-04T12:07:54.627+0000 7f2a7e8db700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi079 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-05-04T12:07:54.946 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[26845]: 2024-05-04T12:07:54.627+0000 7f2a7e8db700 -1 mon.smithi079@0(leader) e1 *** Got Signal Terminated *** 2024-05-04T12:07:54.946 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:54 smithi079 podman[27092]: 2024-05-04 12:07:54.832524949 +0000 UTC m=+0.309376404 container died 182a04754e97359d5f25e57d3914ec7169445f0e49185184688f3217e618e1ff (image=quay.io/ceph/ceph:v18.2.1, name=ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079, org.label-schema.schema-version=1.0, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=-18.2.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.29.1, GIT_BRANCH=HEAD, RELEASE=HEAD) 2024-05-04T12:07:55.335 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 podman[27092]: 2024-05-04 12:07:55.169978312 +0000 UTC m=+0.646829739 container cleanup 182a04754e97359d5f25e57d3914ec7169445f0e49185184688f3217e618e1ff (image=quay.io/ceph/ceph:v18.2.1, name=ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.1, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, io.buildah.version=1.29.1) 2024-05-04T12:07:55.336 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 bash[27092]: ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079 2024-05-04T12:07:55.695 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 podman[27106]: 2024-05-04 12:07:55.337658855 +0000 UTC m=+0.505077729 container remove 182a04754e97359d5f25e57d3914ec7169445f0e49185184688f3217e618e1ff (image=quay.io/ceph/ceph:v18.2.1, name=ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.29.1, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, ceph=True, org.label-schema.schema-version=1.0) 2024-05-04T12:07:55.696 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 systemd[1]: ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968@mon.smithi079.service: Deactivated successfully. 2024-05-04T12:07:55.696 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 systemd[1]: Stopped Ceph mon.smithi079 for d9b985c4-0a0e-11ef-bc97-c7b262605968. 2024-05-04T12:07:55.696 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 systemd[1]: Starting Ceph mon.smithi079 for d9b985c4-0a0e-11ef-bc97-c7b262605968... 2024-05-04T12:07:56.002 INFO:teuthology.orchestra.run.smithi079.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-05-04T12:07:56.017 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 podman[27203]: 2024-05-04 12:07:55.75555814 +0000 UTC m=+0.112815951 container create ce54f11f45dfb16f4c6b4666f6ce4e5e271e8dc289d0e9051cc67cd411322a92 (image=quay.io/ceph/ceph:v18.2.1, name=ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079, io.buildah.version=1.29.1, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222) 2024-05-04T12:07:56.017 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 podman[27203]: 2024-05-04 12:07:55.660610271 +0000 UTC m=+0.017868095 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph:v18.2.1 2024-05-04T12:07:56.017 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 podman[27203]: 2024-05-04 12:07:55.924642789 +0000 UTC m=+0.281900601 container init ce54f11f45dfb16f4c6b4666f6ce4e5e271e8dc289d0e9051cc67cd411322a92 (image=quay.io/ceph/ceph:v18.2.1, name=ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079, ceph=True, CEPH_POINT_RELEASE=-18.2.1, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.29.1, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-05-04T12:07:56.017 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 podman[27203]: 2024-05-04 12:07:55.927438094 +0000 UTC m=+0.284695912 container start ce54f11f45dfb16f4c6b4666f6ce4e5e271e8dc289d0e9051cc67cd411322a92 (image=quay.io/ceph/ceph:v18.2.1, name=ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079, maintainer=Guillaume Abrioux , io.buildah.version=1.29.1, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD) 2024-05-04T12:07:56.017 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: set uid:gid to 167:167 (ceph:ceph) 2024-05-04T12:07:56.017 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable), process ceph-mon, pid 2 2024-05-04T12:07:56.017 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: pidfile_write: ignore empty --pid-file 2024-05-04T12:07:56.017 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: load: jerasure load: lrc 2024-05-04T12:07:56.017 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: RocksDB version: 7.9.2 2024-05-04T12:07:56.017 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Git sha 0 2024-05-04T12:07:56.017 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Compile date 2023-12-11 22:07:34 2024-05-04T12:07:56.017 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: DB SUMMARY 2024-05-04T12:07:56.018 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: DB Session ID: RT6P3D7FCI1X7M2CX2SK 2024-05-04T12:07:56.018 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: CURRENT file: CURRENT 2024-05-04T12:07:56.018 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: IDENTITY file: IDENTITY 2024-05-04T12:07:56.018 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-05-04T12:07:56.018 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi079/store.db dir, Total Num: 1, files: 000008.sst 2024-05-04T12:07:56.018 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi079/store.db: 000009.log size: 98361 ; 2024-05-04T12:07:56.018 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.error_if_exists: 0 2024-05-04T12:07:56.018 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.create_if_missing: 0 2024-05-04T12:07:56.018 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.paranoid_checks: 1 2024-05-04T12:07:56.020 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.flush_verify_memtable_count: 1 2024-05-04T12:07:56.020 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-05-04T12:07:56.020 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-05-04T12:07:56.020 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.env: 0x5607a0cb2720 2024-05-04T12:07:56.020 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.fs: PosixFileSystem 2024-05-04T12:07:56.020 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.info_log: 0x5607a33bb2e0 2024-05-04T12:07:56.020 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.max_file_opening_threads: 16 2024-05-04T12:07:56.020 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.statistics: (nil) 2024-05-04T12:07:56.020 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.use_fsync: 0 2024-05-04T12:07:56.020 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.max_log_file_size: 0 2024-05-04T12:07:56.020 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-05-04T12:07:56.020 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.log_file_time_to_roll: 0 2024-05-04T12:07:56.020 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.keep_log_file_num: 1000 2024-05-04T12:07:56.020 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.recycle_log_file_num: 0 2024-05-04T12:07:56.021 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.allow_fallocate: 1 2024-05-04T12:07:56.021 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.allow_mmap_reads: 0 2024-05-04T12:07:56.021 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.allow_mmap_writes: 0 2024-05-04T12:07:56.021 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.use_direct_reads: 0 2024-05-04T12:07:56.021 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-05-04T12:07:56.021 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.create_missing_column_families: 0 2024-05-04T12:07:56.021 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.db_log_dir: 2024-05-04T12:07:56.021 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.wal_dir: 2024-05-04T12:07:56.021 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.table_cache_numshardbits: 6 2024-05-04T12:07:56.021 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.WAL_ttl_seconds: 0 2024-05-04T12:07:56.021 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.WAL_size_limit_MB: 0 2024-05-04T12:07:56.021 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-05-04T12:07:56.021 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-05-04T12:07:56.021 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.is_fd_close_on_exec: 1 2024-05-04T12:07:56.021 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.advise_random_on_open: 1 2024-05-04T12:07:56.022 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.db_write_buffer_size: 0 2024-05-04T12:07:56.022 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.write_buffer_manager: 0x5607a262c140 2024-05-04T12:07:56.022 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-05-04T12:07:56.022 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-05-04T12:07:56.022 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.use_adaptive_mutex: 0 2024-05-04T12:07:56.022 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.rate_limiter: (nil) 2024-05-04T12:07:56.022 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-05-04T12:07:56.022 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.wal_recovery_mode: 2 2024-05-04T12:07:56.022 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.enable_thread_tracking: 0 2024-05-04T12:07:56.022 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.enable_pipelined_write: 0 2024-05-04T12:07:56.022 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.unordered_write: 0 2024-05-04T12:07:56.022 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-05-04T12:07:56.022 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-05-04T12:07:56.022 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-05-04T12:07:56.023 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-05-04T12:07:56.023 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.row_cache: None 2024-05-04T12:07:56.023 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.wal_filter: None 2024-05-04T12:07:56.023 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-05-04T12:07:56.023 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.allow_ingest_behind: 0 2024-05-04T12:07:56.023 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.two_write_queues: 0 2024-05-04T12:07:56.023 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.manual_wal_flush: 0 2024-05-04T12:07:56.023 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.wal_compression: 0 2024-05-04T12:07:56.023 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.atomic_flush: 0 2024-05-04T12:07:56.023 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-05-04T12:07:56.023 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.persist_stats_to_disk: 0 2024-05-04T12:07:56.023 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.write_dbid_to_manifest: 0 2024-05-04T12:07:56.023 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.log_readahead_size: 0 2024-05-04T12:07:56.023 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-05-04T12:07:56.023 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.best_efforts_recovery: 0 2024-05-04T12:07:56.024 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-05-04T12:07:56.024 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-05-04T12:07:56.024 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.allow_data_in_errors: 0 2024-05-04T12:07:56.024 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.db_host_id: __hostname__ 2024-05-04T12:07:56.024 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.enforce_single_del_contracts: true 2024-05-04T12:07:56.024 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.max_background_jobs: 2 2024-05-04T12:07:56.024 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.max_background_compactions: -1 2024-05-04T12:07:56.024 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.max_subcompactions: 1 2024-05-04T12:07:56.024 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-05-04T12:07:56.024 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-05-04T12:07:56.024 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.delayed_write_rate : 16777216 2024-05-04T12:07:56.024 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.max_total_wal_size: 0 2024-05-04T12:07:56.024 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-05-04T12:07:56.024 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.stats_dump_period_sec: 600 2024-05-04T12:07:56.024 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.stats_persist_period_sec: 600 2024-05-04T12:07:56.025 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-05-04T12:07:56.025 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.max_open_files: -1 2024-05-04T12:07:56.025 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.bytes_per_sync: 0 2024-05-04T12:07:56.025 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.wal_bytes_per_sync: 0 2024-05-04T12:07:56.025 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.strict_bytes_per_sync: 0 2024-05-04T12:07:56.025 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.compaction_readahead_size: 0 2024-05-04T12:07:56.025 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.max_background_flushes: -1 2024-05-04T12:07:56.025 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Compression algorithms supported: 2024-05-04T12:07:56.025 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-05-04T12:07:56.025 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: kZSTD supported: 0 2024-05-04T12:07:56.025 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: kXpressCompression supported: 0 2024-05-04T12:07:56.025 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: kLZ4HCCompression supported: 1 2024-05-04T12:07:56.025 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: kZlibCompression supported: 1 2024-05-04T12:07:56.025 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: kSnappyCompression supported: 1 2024-05-04T12:07:56.026 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: kLZ4Compression supported: 1 2024-05-04T12:07:56.026 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: kBZip2Compression supported: 0 2024-05-04T12:07:56.026 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Fast CRC32 supported: Supported on x86 2024-05-04T12:07:56.026 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: DMutex implementation: pthread_mutex_t 2024-05-04T12:07:56.026 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi079/store.db/MANIFEST-000010 2024-05-04T12:07:56.026 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-05-04T12:07:56.026 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-05-04T12:07:56.027 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.merge_operator: 2024-05-04T12:07:56.027 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.compaction_filter: None 2024-05-04T12:07:56.027 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.compaction_filter_factory: None 2024-05-04T12:07:56.027 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.sst_partitioner_factory: None 2024-05-04T12:07:56.027 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.memtable_factory: SkipListFactory 2024-05-04T12:07:56.027 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.table_factory: BlockBasedTable 2024-05-04T12:07:56.027 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5607a33bb3e0) 2024-05-04T12:07:56.028 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: cache_index_and_filter_blocks: 1 2024-05-04T12:07:56.028 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-05-04T12:07:56.028 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-05-04T12:07:56.028 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: pin_top_level_index_and_filter: 1 2024-05-04T12:07:56.028 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: index_type: 0 2024-05-04T12:07:56.028 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: data_block_index_type: 0 2024-05-04T12:07:56.028 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: index_shortening: 1 2024-05-04T12:07:56.028 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: data_block_hash_table_util_ratio: 0.750000 2024-05-04T12:07:56.028 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: checksum: 4 2024-05-04T12:07:56.028 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: no_block_cache: 0 2024-05-04T12:07:56.028 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: block_cache: 0x5607a26cd350 2024-05-04T12:07:56.028 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: block_cache_name: BinnedLRUCache 2024-05-04T12:07:56.028 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: block_cache_options: 2024-05-04T12:07:56.028 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: capacity : 536870912 2024-05-04T12:07:56.029 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: num_shard_bits : 4 2024-05-04T12:07:56.029 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: strict_capacity_limit : 0 2024-05-04T12:07:56.029 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: high_pri_pool_ratio: 0.000 2024-05-04T12:07:56.029 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: block_cache_compressed: (nil) 2024-05-04T12:07:56.029 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: persistent_cache: (nil) 2024-05-04T12:07:56.029 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: block_size: 4096 2024-05-04T12:07:56.029 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: block_size_deviation: 10 2024-05-04T12:07:56.029 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: block_restart_interval: 16 2024-05-04T12:07:56.029 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: index_block_restart_interval: 1 2024-05-04T12:07:56.029 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: metadata_block_size: 4096 2024-05-04T12:07:56.030 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: partition_filters: 0 2024-05-04T12:07:56.030 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: use_delta_encoding: 1 2024-05-04T12:07:56.030 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: filter_policy: bloomfilter 2024-05-04T12:07:56.030 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: whole_key_filtering: 1 2024-05-04T12:07:56.030 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: verify_compression: 0 2024-05-04T12:07:56.030 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: read_amp_bytes_per_bit: 0 2024-05-04T12:07:56.030 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: format_version: 5 2024-05-04T12:07:56.030 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: enable_index_compression: 1 2024-05-04T12:07:56.030 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: block_align: 0 2024-05-04T12:07:56.030 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: max_auto_readahead_size: 262144 2024-05-04T12:07:56.030 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: prepopulate_block_cache: 0 2024-05-04T12:07:56.030 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: initial_auto_readahead_size: 8192 2024-05-04T12:07:56.030 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: num_file_reads_for_auto_readahead: 2 2024-05-04T12:07:56.030 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.write_buffer_size: 33554432 2024-05-04T12:07:56.031 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.max_write_buffer_number: 2 2024-05-04T12:07:56.031 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.compression: NoCompression 2024-05-04T12:07:56.031 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.bottommost_compression: Disabled 2024-05-04T12:07:56.031 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.prefix_extractor: nullptr 2024-05-04T12:07:56.031 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-05-04T12:07:56.031 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.num_levels: 7 2024-05-04T12:07:56.031 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-05-04T12:07:56.031 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-05-04T12:07:56.031 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-05-04T12:07:56.031 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-05-04T12:07:56.031 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-05-04T12:07:56.031 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-05-04T12:07:56.031 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-05-04T12:07:56.031 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-05-04T12:07:56.031 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-05-04T12:07:56.032 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-05-04T12:07:56.032 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-05-04T12:07:56.032 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-05-04T12:07:56.032 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.compression_opts.window_bits: -14 2024-05-04T12:07:56.032 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.compression_opts.level: 32767 2024-05-04T12:07:56.032 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.compression_opts.strategy: 0 2024-05-04T12:07:56.032 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-05-04T12:07:56.032 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-05-04T12:07:56.032 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-05-04T12:07:56.032 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-05-04T12:07:56.032 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.compression_opts.enabled: false 2024-05-04T12:07:56.032 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-05-04T12:07:56.032 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-05-04T12:07:56.032 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-05-04T12:07:56.032 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-05-04T12:07:56.033 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.target_file_size_base: 67108864 2024-05-04T12:07:56.033 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.target_file_size_multiplier: 1 2024-05-04T12:07:56.033 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-05-04T12:07:56.033 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-05-04T12:07:56.033 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-05-04T12:07:56.033 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-05-04T12:07:56.033 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-05-04T12:07:56.033 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-05-04T12:07:56.033 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-05-04T12:07:56.034 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-05-04T12:07:56.034 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-05-04T12:07:56.034 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-05-04T12:07:56.034 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-05-04T12:07:56.034 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-05-04T12:07:56.034 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-05-04T12:07:56.034 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.arena_block_size: 1048576 2024-05-04T12:07:56.034 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-05-04T12:07:56.034 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-05-04T12:07:56.034 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.disable_auto_compactions: 0 2024-05-04T12:07:56.034 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-05-04T12:07:56.034 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-05-04T12:07:56.034 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-05-04T12:07:56.035 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-05-04T12:07:56.035 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-05-04T12:07:56.035 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-05-04T12:07:56.035 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-05-04T12:07:56.035 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-05-04T12:07:56.035 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-05-04T12:07:56.035 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-05-04T12:07:56.035 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.table_properties_collectors: 2024-05-04T12:07:56.035 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.inplace_update_support: 0 2024-05-04T12:07:56.035 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.inplace_update_num_locks: 10000 2024-05-04T12:07:56.035 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-05-04T12:07:56.035 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-05-04T12:07:56.035 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.memtable_huge_page_size: 0 2024-05-04T12:07:56.035 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.bloom_locality: 0 2024-05-04T12:07:56.035 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.max_successive_merges: 0 2024-05-04T12:07:56.036 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.optimize_filters_for_hits: 0 2024-05-04T12:07:56.036 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.paranoid_file_checks: 0 2024-05-04T12:07:56.036 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.force_consistency_checks: 1 2024-05-04T12:07:56.036 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.report_bg_io_stats: 0 2024-05-04T12:07:56.036 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.ttl: 2592000 2024-05-04T12:07:56.036 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.periodic_compaction_seconds: 0 2024-05-04T12:07:56.036 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-05-04T12:07:56.036 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-05-04T12:07:56.036 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.enable_blob_files: false 2024-05-04T12:07:56.036 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.min_blob_size: 0 2024-05-04T12:07:56.036 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.blob_file_size: 268435456 2024-05-04T12:07:56.036 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.blob_compression_type: NoCompression 2024-05-04T12:07:56.037 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.enable_blob_garbage_collection: false 2024-05-04T12:07:56.037 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-05-04T12:07:56.037 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-05-04T12:07:56.037 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-05-04T12:07:56.037 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.blob_file_starting_level: 0 2024-05-04T12:07:56.037 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-05-04T12:07:56.037 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi079/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-05-04T12:07:56.037 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-05-04T12:07:56.037 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: eacb5fbf-f906-43e0-9493-825e770872db 2024-05-04T12:07:56.037 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714824475955853, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-05-04T12:07:56.037 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-05-04T12:07:56.037 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714824475956846, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 93605, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 314, "table_properties": {"data_size": 91627, "index_size": 260, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13576, "raw_average_key_size": 49, "raw_value_size": 84236, "raw_average_value_size": 308, "num_data_blocks": 11, "num_entries": 273, "num_filter_entries": 273, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1714824475, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "eacb5fbf-f906-43e0-9493-825e770872db", "db_session_id": "RT6P3D7FCI1X7M2CX2SK", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-05-04T12:07:56.037 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714824475956927, "job": 1, "event": "recovery_finished"} 2024-05-04T12:07:56.037 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-05-04T12:07:56.038 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-smithi079/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-05-04T12:07:56.038 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5607a276a000 2024-05-04T12:07:56.038 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: DB pointer 0x5607a2756000 2024-05-04T12:07:56.038 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-05-04T12:07:56.038 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-05-04T12:07:56.038 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: ** DB Stats ** 2024-05-04T12:07:56.038 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-05-04T12:07:56.038 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-05-04T12:07:56.038 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-05-04T12:07:56.038 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-05-04T12:07:56.038 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-05-04T12:07:56.038 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-05-04T12:07:56.038 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-05-04T12:07:56.039 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: 2024-05-04T12:07:56.039 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: ** Compaction Stats [default] ** 2024-05-04T12:07:56.039 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-05-04T12:07:56.039 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-05-04T12:07:56.039 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: L0 2/0 93.28 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 143.5 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-05-04T12:07:56.039 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: Sum 2/0 93.28 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 143.5 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-05-04T12:07:56.039 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 143.5 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-05-04T12:07:56.039 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: 2024-05-04T12:07:56.039 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: ** Compaction Stats [default] ** 2024-05-04T12:07:56.039 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-05-04T12:07:56.039 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-05-04T12:07:56.039 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 143.5 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-05-04T12:07:56.039 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: 2024-05-04T12:07:56.040 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-05-04T12:07:56.040 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: 2024-05-04T12:07:56.040 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-05-04T12:07:56.040 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-05-04T12:07:56.040 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-05-04T12:07:56.040 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-05-04T12:07:56.040 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-05-04T12:07:56.040 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: AddFile(Keys): cumulative 0, interval 0 2024-05-04T12:07:56.040 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: Cumulative compaction: 0.00 GB write, 30.76 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-04T12:07:56.040 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: Interval compaction: 0.00 GB write, 30.76 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-04T12:07:56.040 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-05-04T12:07:56.040 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: Block cache BinnedLRUCache@0x5607a26cd350#2 capacity: 512.00 MB usage: 1.36 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2024-05-04T12:07:56.040 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.89 KB,0.000169873%) IndexBlock(2,0.47 KB,8.9407e-05%) Misc(1,0.00 KB,0%) 2024-05-04T12:07:56.040 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: 2024-05-04T12:07:56.040 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: ** File Read Latency Histogram By Level [default] ** 2024-05-04T12:07:56.041 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: starting mon.smithi079 rank 0 at public addrs [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] at bind addrs [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi079 fsid d9b985c4-0a0e-11ef-bc97-c7b262605968 2024-05-04T12:07:56.041 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: mon.smithi079@-1(???) e1 preinit fsid d9b985c4-0a0e-11ef-bc97-c7b262605968 2024-05-04T12:07:56.041 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: mon.smithi079@-1(???).mds e1 new map 2024-05-04T12:07:56.041 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: mon.smithi079@-1(???).mds e1 print_map 2024-05-04T12:07:56.041 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: e1 2024-05-04T12:07:56.041 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-05-04T12:07:56.041 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-05-04T12:07:56.041 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: legacy client fscid: -1 2024-05-04T12:07:56.041 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: 2024-05-04T12:07:56.041 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: No filesystems configured 2024-05-04T12:07:56.041 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: mon.smithi079@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-05-04T12:07:56.041 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: mon.smithi079@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-05-04T12:07:56.041 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: mon.smithi079@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-05-04T12:07:56.041 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: mon.smithi079@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-05-04T12:07:56.042 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: mon.smithi079@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-05-04T12:07:56.042 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: expand_channel_meta expand map: {default=false} 2024-05-04T12:07:56.042 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: expand_channel_meta from 'false' to 'false' 2024-05-04T12:07:56.042 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: expand_channel_meta expanded map: {default=false} 2024-05-04T12:07:56.042 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: expand_channel_meta expand map: {default=info} 2024-05-04T12:07:56.042 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: expand_channel_meta from 'info' to 'info' 2024-05-04T12:07:56.042 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: expand_channel_meta expanded map: {default=info} 2024-05-04T12:07:56.042 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: expand_channel_meta expand map: {default=daemon} 2024-05-04T12:07:56.042 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: expand_channel_meta from 'daemon' to 'daemon' 2024-05-04T12:07:56.042 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: expand_channel_meta expanded map: {default=daemon} 2024-05-04T12:07:56.042 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: expand_channel_meta expand map: {default=debug} 2024-05-04T12:07:56.042 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: expand_channel_meta from 'debug' to 'debug' 2024-05-04T12:07:56.042 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 ceph-mon[27222]: expand_channel_meta expanded map: {default=debug} 2024-05-04T12:07:56.042 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:55 smithi079 bash[27203]: ce54f11f45dfb16f4c6b4666f6ce4e5e271e8dc289d0e9051cc67cd411322a92 2024-05-04T12:07:56.043 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:56 smithi079 systemd[1]: Started Ceph mon.smithi079 for d9b985c4-0a0e-11ef-bc97-c7b262605968. 2024-05-04T12:07:56.043 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:56 smithi079 ceph-mon[27222]: mon.smithi079 is new leader, mons smithi079 in quorum (ranks 0) 2024-05-04T12:07:56.043 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:56 smithi079 ceph-mon[27222]: monmap e1: 1 mons at {smithi079=[v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-05-04T12:07:56.043 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:56 smithi079 ceph-mon[27222]: fsmap 2024-05-04T12:07:56.043 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:56 smithi079 ceph-mon[27222]: osdmap e1: 0 total, 0 up, 0 in 2024-05-04T12:07:56.043 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:56 smithi079 ceph-mon[27222]: mgrmap e1: no daemons active 2024-05-04T12:07:57.054 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:56.446+0000 7fbed4c6c700 1 Processor -- start 2024-05-04T12:07:57.054 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:56.446+0000 7fbed4c6c700 1 -- start start 2024-05-04T12:07:57.054 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:56.447+0000 7fbed4c6c700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbed0106560 0x7fbed0106980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:07:57.054 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:56.447+0000 7fbed4c6c700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbed0106f50 con 0x7fbed0106560 2024-05-04T12:07:57.054 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:56.447+0000 7fbece59c700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbed0106560 0x7fbed0106980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:07:57.054 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:56.447+0000 7fbece59c700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbed0106560 0x7fbed0106980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:44876/0 (socket says 172.21.15.79:44876) 2024-05-04T12:07:57.055 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:56.447+0000 7fbece59c700 1 -- 172.21.15.79:0/2966134352 learned_addr learned my addr 172.21.15.79:0/2966134352 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:07:57.055 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:56.447+0000 7fbece59c700 1 -- 172.21.15.79:0/2966134352 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbed0107760 con 0x7fbed0106560 2024-05-04T12:07:57.055 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:56.447+0000 7fbece59c700 1 --2- 172.21.15.79:0/2966134352 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbed0106560 0x7fbed0106980 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fbec00098a0 tx=0x7fbec0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c85f74cac47fd4c server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:07:57.055 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:56.448+0000 7fbecd59a700 1 -- 172.21.15.79:0/2966134352 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbec0004180 con 0x7fbed0106560 2024-05-04T12:07:57.055 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:56.448+0000 7fbecd59a700 1 -- 172.21.15.79:0/2966134352 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fbec00042e0 con 0x7fbed0106560 2024-05-04T12:07:57.055 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:56.448+0000 7fbecd59a700 1 -- 172.21.15.79:0/2966134352 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbec0011a50 con 0x7fbed0106560 2024-05-04T12:07:57.055 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:56.448+0000 7fbed4c6c700 1 -- 172.21.15.79:0/2966134352 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbed0106560 msgr2=0x7fbed0106980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:07:57.055 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:56.448+0000 7fbed4c6c700 1 --2- 172.21.15.79:0/2966134352 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbed0106560 0x7fbed0106980 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fbec00098a0 tx=0x7fbec0009bb0 comp rx=0 tx=0).stop 2024-05-04T12:07:57.055 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:56.448+0000 7fbed4c6c700 1 -- 172.21.15.79:0/2966134352 shutdown_connections 2024-05-04T12:07:57.055 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:56.448+0000 7fbed4c6c700 1 --2- 172.21.15.79:0/2966134352 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbed0106560 0x7fbed0106980 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:07:57.055 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:56.448+0000 7fbed4c6c700 1 -- 172.21.15.79:0/2966134352 >> 172.21.15.79:0/2966134352 conn(0x7fbed0101b20 msgr2=0x7fbed0103f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:07:57.055 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:56.448+0000 7fbed4c6c700 1 -- 172.21.15.79:0/2966134352 shutdown_connections 2024-05-04T12:07:57.055 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:56.448+0000 7fbed4c6c700 1 -- 172.21.15.79:0/2966134352 wait complete. 2024-05-04T12:07:57.055 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:56.449+0000 7fbed4c6c700 1 Processor -- start 2024-05-04T12:07:57.056 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:56.449+0000 7fbed4c6c700 1 -- start start 2024-05-04T12:07:57.056 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:56.449+0000 7fbed4c6c700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbed0106560 0x7fbed019a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:07:57.056 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:56.449+0000 7fbed4c6c700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbed019a6e0 con 0x7fbed0106560 2024-05-04T12:07:57.056 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:56.449+0000 7fbece59c700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbed0106560 0x7fbed019a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:07:57.056 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:56.449+0000 7fbece59c700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbed0106560 0x7fbed019a1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:44880/0 (socket says 172.21.15.79:44880) 2024-05-04T12:07:57.056 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:56.449+0000 7fbece59c700 1 -- 172.21.15.79:0/3113636835 learned_addr learned my addr 172.21.15.79:0/3113636835 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:07:57.056 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:56.450+0000 7fbece59c700 1 -- 172.21.15.79:0/3113636835 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbec0009550 con 0x7fbed0106560 2024-05-04T12:07:57.056 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:56.450+0000 7fbece59c700 1 --2- 172.21.15.79:0/3113636835 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbed0106560 0x7fbed019a1a0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fbec0006af0 tx=0x7fbec0011a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:07:57.056 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:56.450+0000 7fbebf7fe700 1 -- 172.21.15.79:0/3113636835 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbec0003b90 con 0x7fbed0106560 2024-05-04T12:07:57.056 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:56.450+0000 7fbebf7fe700 1 -- 172.21.15.79:0/3113636835 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fbec001a8a0 con 0x7fbed0106560 2024-05-04T12:07:57.056 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:56.450+0000 7fbed4c6c700 1 -- 172.21.15.79:0/3113636835 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbed019a8e0 con 0x7fbed0106560 2024-05-04T12:07:57.056 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:56.450+0000 7fbebf7fe700 1 -- 172.21.15.79:0/3113636835 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbec0019880 con 0x7fbed0106560 2024-05-04T12:07:57.056 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:56.450+0000 7fbed4c6c700 1 -- 172.21.15.79:0/3113636835 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbed019ad80 con 0x7fbed0106560 2024-05-04T12:07:57.056 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:56.450+0000 7fbebf7fe700 1 -- 172.21.15.79:0/3113636835 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fbec002d440 con 0x7fbed0106560 2024-05-04T12:07:57.057 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:56.450+0000 7fbebf7fe700 1 -- 172.21.15.79:0/3113636835 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fbec0024860 con 0x7fbed0106560 2024-05-04T12:07:57.057 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:56.451+0000 7fbed4c6c700 1 -- 172.21.15.79:0/3113636835 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbed00623c0 con 0x7fbed0106560 2024-05-04T12:07:57.057 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:56.453+0000 7fbebf7fe700 1 -- 172.21.15.79:0/3113636835 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7fbec001aa10 con 0x7fbed0106560 2024-05-04T12:07:57.057 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:56.519+0000 7fbed4c6c700 1 -- 172.21.15.79:0/3113636835 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7fbed010a980 con 0x7fbed0106560 2024-05-04T12:07:57.057 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:56.524+0000 7fbebf7fe700 1 -- 172.21.15.79:0/3113636835 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7fbec001abb0 con 0x7fbed0106560 2024-05-04T12:07:57.057 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:56.525+0000 7fbed4c6c700 1 -- 172.21.15.79:0/3113636835 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbed0106560 msgr2=0x7fbed019a1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:07:57.057 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:56.525+0000 7fbed4c6c700 1 --2- 172.21.15.79:0/3113636835 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbed0106560 0x7fbed019a1a0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fbec0006af0 tx=0x7fbec0011a90 comp rx=0 tx=0).stop 2024-05-04T12:07:57.057 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:56.525+0000 7fbed4c6c700 1 -- 172.21.15.79:0/3113636835 shutdown_connections 2024-05-04T12:07:57.057 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:56.525+0000 7fbed4c6c700 1 --2- 172.21.15.79:0/3113636835 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbed0106560 0x7fbed019a1a0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:07:57.057 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:56.525+0000 7fbed4c6c700 1 -- 172.21.15.79:0/3113636835 >> 172.21.15.79:0/3113636835 conn(0x7fbed0101b20 msgr2=0x7fbed0102840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:07:57.057 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:56.525+0000 7fbed4c6c700 1 -- 172.21.15.79:0/3113636835 shutdown_connections 2024-05-04T12:07:57.057 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:56.525+0000 7fbed4c6c700 1 -- 172.21.15.79:0/3113636835 wait complete. 2024-05-04T12:07:57.057 INFO:teuthology.orchestra.run.smithi079.stdout:Wrote config to /etc/ceph/ceph.conf 2024-05-04T12:07:57.080 INFO:teuthology.orchestra.run.smithi079.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-05-04T12:07:57.080 INFO:teuthology.orchestra.run.smithi079.stdout:Creating mgr... 2024-05-04T12:07:57.080 INFO:teuthology.orchestra.run.smithi079.stdout:Verifying port 0.0.0.0:9283 ... 2024-05-04T12:07:57.080 INFO:teuthology.orchestra.run.smithi079.stdout:Verifying port 0.0.0.0:8765 ... 2024-05-04T12:07:57.081 INFO:teuthology.orchestra.run.smithi079.stdout:Verifying port 0.0.0.0:8443 ... 2024-05-04T12:07:57.324 INFO:teuthology.orchestra.run.smithi079.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968@mgr.smithi079.dgcpwl 2024-05-04T12:07:57.324 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stderr Failed to reset failed state of unit ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968@mgr.smithi079.dgcpwl.service: Unit ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968@mgr.smithi079.dgcpwl.service not loaded. 2024-05-04T12:07:57.469 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968.target.wants/ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968@mgr.smithi079.dgcpwl.service → /etc/systemd/system/ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968@.service. 2024-05-04T12:07:57.794 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:57 smithi079 ceph-mon[27222]: from='client.? 172.21.15.79:0/3113636835' entity='client.admin' 2024-05-04T12:07:57.981 INFO:teuthology.orchestra.run.smithi079.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-05-04T12:07:57.982 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stdout disabled 2024-05-04T12:07:57.988 INFO:teuthology.orchestra.run.smithi079.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-05-04T12:07:57.989 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stdout inactive 2024-05-04T12:07:57.989 INFO:teuthology.orchestra.run.smithi079.stdout:firewalld.service is not enabled 2024-05-04T12:07:57.989 INFO:teuthology.orchestra.run.smithi079.stdout:Not possible to enable service . firewalld.service is not available 2024-05-04T12:07:57.997 INFO:teuthology.orchestra.run.smithi079.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-05-04T12:07:57.997 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stdout disabled 2024-05-04T12:07:58.004 INFO:teuthology.orchestra.run.smithi079.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-05-04T12:07:58.004 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stdout inactive 2024-05-04T12:07:58.005 INFO:teuthology.orchestra.run.smithi079.stdout:firewalld.service is not enabled 2024-05-04T12:07:58.005 INFO:teuthology.orchestra.run.smithi079.stdout:Not possible to open ports <[9283, 8765, 8443]>. firewalld.service is not available 2024-05-04T12:07:58.005 INFO:teuthology.orchestra.run.smithi079.stdout:Waiting for mgr to start... 2024-05-04T12:07:58.005 INFO:teuthology.orchestra.run.smithi079.stdout:Waiting for mgr... 2024-05-04T12:07:58.638 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:07:58 smithi079 ceph-mon[27222]: from='client.? 172.21.15.79:0/3277686842' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-05-04T12:07:59.056 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout 2024-05-04T12:07:59.056 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout { 2024-05-04T12:07:59.056 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "fsid": "d9b985c4-0a0e-11ef-bc97-c7b262605968", 2024-05-04T12:07:59.056 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "health": { 2024-05-04T12:07:59.056 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-05-04T12:07:59.056 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-05-04T12:07:59.056 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-05-04T12:07:59.057 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-05-04T12:07:59.057 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-05-04T12:07:59.057 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-05-04T12:07:59.057 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout 0 2024-05-04T12:07:59.057 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout ], 2024-05-04T12:07:59.057 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-05-04T12:07:59.057 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "smithi079" 2024-05-04T12:07:59.057 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout ], 2024-05-04T12:07:59.057 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-05-04T12:07:59.057 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "monmap": { 2024-05-04T12:07:59.057 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-04T12:07:59.057 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-05-04T12:07:59.057 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-05-04T12:07:59.057 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-05-04T12:07:59.057 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-05-04T12:07:59.058 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-04T12:07:59.058 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-05-04T12:07:59.058 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-05-04T12:07:59.058 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-05-04T12:07:59.066 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-05-04T12:07:59.066 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-05-04T12:07:59.066 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-05-04T12:07:59.066 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-05-04T12:07:59.066 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-05-04T12:07:59.066 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-05-04T12:07:59.066 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-05-04T12:07:59.066 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-05-04T12:07:59.066 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-05-04T12:07:59.066 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-05-04T12:07:59.066 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-05-04T12:07:59.066 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-05-04T12:07:59.066 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-05-04T12:07:59.067 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-05-04T12:07:59.067 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-05-04T12:07:59.067 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-04T12:07:59.067 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-05-04T12:07:59.067 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-05-04T12:07:59.067 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-05-04T12:07:59.067 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-05-04T12:07:59.067 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "available": false, 2024-05-04T12:07:59.067 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-05-04T12:07:59.067 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "modules": [ 2024-05-04T12:07:59.067 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "iostat", 2024-05-04T12:07:59.067 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "nfs", 2024-05-04T12:07:59.067 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "restful" 2024-05-04T12:07:59.067 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout ], 2024-05-04T12:07:59.068 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-04T12:07:59.068 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-05-04T12:07:59.068 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-05-04T12:07:59.068 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-04T12:07:59.068 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "modified": "2024-05-04T12:07:51.285519+0000", 2024-05-04T12:07:59.068 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-04T12:07:59.068 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-05-04T12:07:59.068 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-05-04T12:07:59.068 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout } 2024-05-04T12:07:59.068 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:58.414+0000 7f421efa1700 1 Processor -- start 2024-05-04T12:07:59.068 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:58.414+0000 7f421efa1700 1 -- start start 2024-05-04T12:07:59.068 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:58.415+0000 7f421efa1700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f421807be60 0x7f421807a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:07:59.068 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:58.415+0000 7f421efa1700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f421807a890 con 0x7f421807be60 2024-05-04T12:07:59.068 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:58.415+0000 7f421df9f700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f421807be60 0x7f421807a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:07:59.068 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:58.415+0000 7f421df9f700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f421807be60 0x7f421807a2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:44898/0 (socket says 172.21.15.79:44898) 2024-05-04T12:07:59.069 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:58.415+0000 7f421df9f700 1 -- 172.21.15.79:0/4103122005 learned_addr learned my addr 172.21.15.79:0/4103122005 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:07:59.069 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:58.415+0000 7f421df9f700 1 -- 172.21.15.79:0/4103122005 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f421807a9d0 con 0x7f421807be60 2024-05-04T12:07:59.069 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:58.415+0000 7f421df9f700 1 --2- 172.21.15.79:0/4103122005 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f421807be60 0x7f421807a2c0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f42140098a0 tx=0x7f4214009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2409d69b73aa43b7 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:07:59.069 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:58.415+0000 7f421cf9d700 1 -- 172.21.15.79:0/4103122005 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4214004180 con 0x7f421807be60 2024-05-04T12:07:59.069 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:58.415+0000 7f421cf9d700 1 -- 172.21.15.79:0/4103122005 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f42140042e0 con 0x7f421807be60 2024-05-04T12:07:59.069 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:58.415+0000 7f421cf9d700 1 -- 172.21.15.79:0/4103122005 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4214011a50 con 0x7f421807be60 2024-05-04T12:07:59.069 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:58.416+0000 7f421efa1700 1 -- 172.21.15.79:0/4103122005 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f421807be60 msgr2=0x7f421807a2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:07:59.069 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:58.416+0000 7f421efa1700 1 --2- 172.21.15.79:0/4103122005 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f421807be60 0x7f421807a2c0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f42140098a0 tx=0x7f4214009bb0 comp rx=0 tx=0).stop 2024-05-04T12:07:59.069 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:58.416+0000 7f421efa1700 1 -- 172.21.15.79:0/4103122005 shutdown_connections 2024-05-04T12:07:59.069 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:58.416+0000 7f421efa1700 1 --2- 172.21.15.79:0/4103122005 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f421807be60 0x7f421807a2c0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:07:59.069 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:58.416+0000 7f421efa1700 1 -- 172.21.15.79:0/4103122005 >> 172.21.15.79:0/4103122005 conn(0x7f4218101980 msgr2=0x7f4218103dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:07:59.069 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:58.416+0000 7f421efa1700 1 -- 172.21.15.79:0/4103122005 shutdown_connections 2024-05-04T12:07:59.069 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:58.416+0000 7f421efa1700 1 -- 172.21.15.79:0/4103122005 wait complete. 2024-05-04T12:07:59.069 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:58.417+0000 7f421efa1700 1 Processor -- start 2024-05-04T12:07:59.069 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:58.417+0000 7f421efa1700 1 -- start start 2024-05-04T12:07:59.070 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:58.417+0000 7f421efa1700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f421807be60 0x7f421819c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:07:59.070 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:58.417+0000 7f421efa1700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f421819c7d0 con 0x7f421807be60 2024-05-04T12:07:59.070 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:58.417+0000 7f421df9f700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f421807be60 0x7f421819c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:07:59.070 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:58.417+0000 7f421df9f700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f421807be60 0x7f421819c290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:44904/0 (socket says 172.21.15.79:44904) 2024-05-04T12:07:59.070 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:58.417+0000 7f421df9f700 1 -- 172.21.15.79:0/3277686842 learned_addr learned my addr 172.21.15.79:0/3277686842 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:07:59.070 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:58.417+0000 7f421df9f700 1 -- 172.21.15.79:0/3277686842 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4214009550 con 0x7f421807be60 2024-05-04T12:07:59.070 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:58.417+0000 7f421df9f700 1 --2- 172.21.15.79:0/3277686842 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f421807be60 0x7f421819c290 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f4214003f80 tx=0x7f4214011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:07:59.070 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:58.417+0000 7f420effd700 1 -- 172.21.15.79:0/3277686842 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4214003a80 con 0x7f421807be60 2024-05-04T12:07:59.070 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:58.417+0000 7f421efa1700 1 -- 172.21.15.79:0/3277686842 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f421819c9d0 con 0x7f421807be60 2024-05-04T12:07:59.070 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:58.418+0000 7f420effd700 1 -- 172.21.15.79:0/3277686842 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f4214003be0 con 0x7f421807be60 2024-05-04T12:07:59.070 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:58.418+0000 7f420effd700 1 -- 172.21.15.79:0/3277686842 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4214019820 con 0x7f421807be60 2024-05-04T12:07:59.070 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:58.418+0000 7f421efa1700 1 -- 172.21.15.79:0/3277686842 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f421819ce70 con 0x7f421807be60 2024-05-04T12:07:59.070 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:58.418+0000 7f420effd700 1 -- 172.21.15.79:0/3277686842 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f421402d440 con 0x7f421807be60 2024-05-04T12:07:59.071 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:58.418+0000 7f420effd700 1 -- 172.21.15.79:0/3277686842 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f4214024810 con 0x7f421807be60 2024-05-04T12:07:59.071 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:58.418+0000 7f421efa1700 1 -- 172.21.15.79:0/3277686842 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42180623c0 con 0x7f421807be60 2024-05-04T12:07:59.071 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:58.420+0000 7f420effd700 1 -- 172.21.15.79:0/3277686842 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7f421401acc0 con 0x7f421807be60 2024-05-04T12:07:59.071 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:58.492+0000 7f421efa1700 1 -- 172.21.15.79:0/3277686842 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f421819f920 con 0x7f421807be60 2024-05-04T12:07:59.071 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:58.492+0000 7f420effd700 1 -- 172.21.15.79:0/3277686842 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1246 (secure 0 0 0) 0x7f4214024b20 con 0x7f421807be60 2024-05-04T12:07:59.071 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:58.493+0000 7f421efa1700 1 -- 172.21.15.79:0/3277686842 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f421807be60 msgr2=0x7f421819c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:07:59.071 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:58.493+0000 7f421efa1700 1 --2- 172.21.15.79:0/3277686842 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f421807be60 0x7f421819c290 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f4214003f80 tx=0x7f4214011ab0 comp rx=0 tx=0).stop 2024-05-04T12:07:59.071 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:58.493+0000 7f421efa1700 1 -- 172.21.15.79:0/3277686842 shutdown_connections 2024-05-04T12:07:59.071 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:58.493+0000 7f421efa1700 1 --2- 172.21.15.79:0/3277686842 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f421807be60 0x7f421819c290 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:07:59.071 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:58.493+0000 7f421efa1700 1 -- 172.21.15.79:0/3277686842 >> 172.21.15.79:0/3277686842 conn(0x7f4218101980 msgr2=0x7f4218102ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:07:59.071 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:58.493+0000 7f421efa1700 1 -- 172.21.15.79:0/3277686842 shutdown_connections 2024-05-04T12:07:59.071 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:07:58.493+0000 7f421efa1700 1 -- 172.21.15.79:0/3277686842 wait complete. 2024-05-04T12:07:59.071 INFO:teuthology.orchestra.run.smithi079.stdout:mgr not available, waiting (1/15)... 2024-05-04T12:08:02.251 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:08:01 smithi079 ceph-mon[27222]: from='client.? 172.21.15.79:0/1934945055' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-05-04T12:08:02.256 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout 2024-05-04T12:08:02.256 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout { 2024-05-04T12:08:02.256 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "fsid": "d9b985c4-0a0e-11ef-bc97-c7b262605968", 2024-05-04T12:08:02.256 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "health": { 2024-05-04T12:08:02.256 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-05-04T12:08:02.256 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-05-04T12:08:02.257 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-05-04T12:08:02.257 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-05-04T12:08:02.257 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-05-04T12:08:02.257 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-05-04T12:08:02.257 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout 0 2024-05-04T12:08:02.257 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout ], 2024-05-04T12:08:02.257 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-05-04T12:08:02.257 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "smithi079" 2024-05-04T12:08:02.257 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout ], 2024-05-04T12:08:02.257 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-05-04T12:08:02.257 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "monmap": { 2024-05-04T12:08:02.257 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-04T12:08:02.257 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-05-04T12:08:02.257 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-05-04T12:08:02.258 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-05-04T12:08:02.258 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-05-04T12:08:02.258 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-04T12:08:02.258 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-05-04T12:08:02.258 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-05-04T12:08:02.258 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-05-04T12:08:02.266 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-05-04T12:08:02.266 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-05-04T12:08:02.266 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-05-04T12:08:02.266 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-05-04T12:08:02.266 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-05-04T12:08:02.266 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-05-04T12:08:02.266 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-05-04T12:08:02.266 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-05-04T12:08:02.266 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-05-04T12:08:02.266 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-05-04T12:08:02.267 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-05-04T12:08:02.267 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-05-04T12:08:02.267 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-05-04T12:08:02.267 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-05-04T12:08:02.267 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-05-04T12:08:02.267 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-04T12:08:02.267 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-05-04T12:08:02.267 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-05-04T12:08:02.267 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-05-04T12:08:02.267 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-05-04T12:08:02.267 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "available": false, 2024-05-04T12:08:02.267 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-05-04T12:08:02.267 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "modules": [ 2024-05-04T12:08:02.267 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "iostat", 2024-05-04T12:08:02.268 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "nfs", 2024-05-04T12:08:02.268 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "restful" 2024-05-04T12:08:02.268 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout ], 2024-05-04T12:08:02.268 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-04T12:08:02.268 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-05-04T12:08:02.268 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-05-04T12:08:02.268 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-04T12:08:02.268 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "modified": "2024-05-04T12:07:51.285519+0000", 2024-05-04T12:08:02.268 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-04T12:08:02.268 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-05-04T12:08:02.268 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-05-04T12:08:02.268 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout } 2024-05-04T12:08:02.268 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:01.499+0000 7f03cf949700 1 Processor -- start 2024-05-04T12:08:02.268 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:01.499+0000 7f03cf949700 1 -- start start 2024-05-04T12:08:02.269 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:01.499+0000 7f03cf949700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f03c8104860 0x7f03c8104c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:08:02.269 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:01.499+0000 7f03cf949700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03c8105250 con 0x7f03c8104860 2024-05-04T12:08:02.269 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:01.500+0000 7f03cd6e5700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f03c8104860 0x7f03c8104c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:08:02.269 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:01.500+0000 7f03cd6e5700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f03c8104860 0x7f03c8104c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:44906/0 (socket says 172.21.15.79:44906) 2024-05-04T12:08:02.269 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:01.500+0000 7f03cd6e5700 1 -- 172.21.15.79:0/2991334768 learned_addr learned my addr 172.21.15.79:0/2991334768 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:08:02.269 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:01.500+0000 7f03cd6e5700 1 -- 172.21.15.79:0/2991334768 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03c81077f0 con 0x7f03c8104860 2024-05-04T12:08:02.269 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:01.500+0000 7f03cd6e5700 1 --2- 172.21.15.79:0/2991334768 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f03c8104860 0x7f03c8104c80 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f03c40098a0 tx=0x7f03c4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7373fb31b93bb74 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:08:02.269 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:01.500+0000 7f03bffff700 1 -- 172.21.15.79:0/2991334768 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f03c4004180 con 0x7f03c8104860 2024-05-04T12:08:02.269 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:01.500+0000 7f03bffff700 1 -- 172.21.15.79:0/2991334768 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f03c40042e0 con 0x7f03c8104860 2024-05-04T12:08:02.269 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:01.500+0000 7f03bffff700 1 -- 172.21.15.79:0/2991334768 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f03c4011a50 con 0x7f03c8104860 2024-05-04T12:08:02.269 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:01.501+0000 7f03cf949700 1 -- 172.21.15.79:0/2991334768 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f03c8104860 msgr2=0x7f03c8104c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:08:02.269 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:01.501+0000 7f03cf949700 1 --2- 172.21.15.79:0/2991334768 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f03c8104860 0x7f03c8104c80 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f03c40098a0 tx=0x7f03c4009bb0 comp rx=0 tx=0).stop 2024-05-04T12:08:02.269 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:01.501+0000 7f03cf949700 1 -- 172.21.15.79:0/2991334768 shutdown_connections 2024-05-04T12:08:02.269 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:01.501+0000 7f03cf949700 1 --2- 172.21.15.79:0/2991334768 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f03c8104860 0x7f03c8104c80 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:08:02.270 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:01.501+0000 7f03cf949700 1 -- 172.21.15.79:0/2991334768 >> 172.21.15.79:0/2991334768 conn(0x7f03c8100190 msgr2=0x7f03c81025d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:08:02.270 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:01.501+0000 7f03cf949700 1 -- 172.21.15.79:0/2991334768 shutdown_connections 2024-05-04T12:08:02.270 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:01.501+0000 7f03cf949700 1 -- 172.21.15.79:0/2991334768 wait complete. 2024-05-04T12:08:02.270 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:01.502+0000 7f03cf949700 1 Processor -- start 2024-05-04T12:08:02.270 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:01.502+0000 7f03cf949700 1 -- start start 2024-05-04T12:08:02.270 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:01.502+0000 7f03cf949700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f03c8104860 0x7f03c819a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:08:02.270 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:01.502+0000 7f03cf949700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03c819a720 con 0x7f03c8104860 2024-05-04T12:08:02.270 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:01.502+0000 7f03cd6e5700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f03c8104860 0x7f03c819a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:08:02.270 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:01.502+0000 7f03cd6e5700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f03c8104860 0x7f03c819a1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:44910/0 (socket says 172.21.15.79:44910) 2024-05-04T12:08:02.270 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:01.503+0000 7f03cd6e5700 1 -- 172.21.15.79:0/1934945055 learned_addr learned my addr 172.21.15.79:0/1934945055 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:08:02.270 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:01.503+0000 7f03cd6e5700 1 -- 172.21.15.79:0/1934945055 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03c4009550 con 0x7f03c8104860 2024-05-04T12:08:02.270 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:01.503+0000 7f03cd6e5700 1 --2- 172.21.15.79:0/1934945055 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f03c8104860 0x7f03c819a1e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f03c4006af0 tx=0x7f03c4011a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:08:02.270 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:01.503+0000 7f03be7fc700 1 -- 172.21.15.79:0/1934945055 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f03c4003b90 con 0x7f03c8104860 2024-05-04T12:08:02.271 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:01.503+0000 7f03cf949700 1 -- 172.21.15.79:0/1934945055 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03c819a920 con 0x7f03c8104860 2024-05-04T12:08:02.271 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:01.503+0000 7f03be7fc700 1 -- 172.21.15.79:0/1934945055 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f03c401a8a0 con 0x7f03c8104860 2024-05-04T12:08:02.271 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:01.503+0000 7f03be7fc700 1 -- 172.21.15.79:0/1934945055 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f03c4019880 con 0x7f03c8104860 2024-05-04T12:08:02.271 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:01.503+0000 7f03cf949700 1 -- 172.21.15.79:0/1934945055 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03c819adc0 con 0x7f03c8104860 2024-05-04T12:08:02.271 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:01.504+0000 7f03be7fc700 1 -- 172.21.15.79:0/1934945055 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f03c402d440 con 0x7f03c8104860 2024-05-04T12:08:02.271 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:01.504+0000 7f03be7fc700 1 -- 172.21.15.79:0/1934945055 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f03c4024860 con 0x7f03c8104860 2024-05-04T12:08:02.271 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:01.504+0000 7f03cf949700 1 -- 172.21.15.79:0/1934945055 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03c80623c0 con 0x7f03c8104860 2024-05-04T12:08:02.271 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:01.508+0000 7f03be7fc700 1 -- 172.21.15.79:0/1934945055 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7f03c401aa10 con 0x7f03c8104860 2024-05-04T12:08:02.271 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:01.581+0000 7f03cf949700 1 -- 172.21.15.79:0/1934945055 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f03c810aaa0 con 0x7f03c8104860 2024-05-04T12:08:02.271 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:01.582+0000 7f03be7fc700 1 -- 172.21.15.79:0/1934945055 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1246 (secure 0 0 0) 0x7f03c401abb0 con 0x7f03c8104860 2024-05-04T12:08:02.271 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:01.583+0000 7f03cf949700 1 -- 172.21.15.79:0/1934945055 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f03c8104860 msgr2=0x7f03c819a1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:08:02.271 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:01.583+0000 7f03cf949700 1 --2- 172.21.15.79:0/1934945055 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f03c8104860 0x7f03c819a1e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f03c4006af0 tx=0x7f03c4011a90 comp rx=0 tx=0).stop 2024-05-04T12:08:02.271 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:01.583+0000 7f03cf949700 1 -- 172.21.15.79:0/1934945055 shutdown_connections 2024-05-04T12:08:02.271 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:01.583+0000 7f03cf949700 1 --2- 172.21.15.79:0/1934945055 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f03c8104860 0x7f03c819a1e0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:08:02.271 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:01.583+0000 7f03cf949700 1 -- 172.21.15.79:0/1934945055 >> 172.21.15.79:0/1934945055 conn(0x7f03c8100190 msgr2=0x7f03c8100f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:08:02.272 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:01.583+0000 7f03cf949700 1 -- 172.21.15.79:0/1934945055 shutdown_connections 2024-05-04T12:08:02.272 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:01.583+0000 7f03cf949700 1 -- 172.21.15.79:0/1934945055 wait complete. 2024-05-04T12:08:02.272 INFO:teuthology.orchestra.run.smithi079.stdout:mgr not available, waiting (2/15)... 2024-05-04T12:08:04.945 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:08:04 smithi079 ceph-mon[27222]: Activating manager daemon smithi079.dgcpwl 2024-05-04T12:08:04.945 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:08:04 smithi079 ceph-mon[27222]: mgrmap e2: smithi079.dgcpwl(active, starting, since 0.00279802s) 2024-05-04T12:08:04.945 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:08:04 smithi079 ceph-mon[27222]: from='mgr.14100 172.21.15.79:0/3968367628' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-04T12:08:04.945 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:08:04 smithi079 ceph-mon[27222]: from='mgr.14100 172.21.15.79:0/3968367628' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-04T12:08:04.945 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:08:04 smithi079 ceph-mon[27222]: from='mgr.14100 172.21.15.79:0/3968367628' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-04T12:08:04.945 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:08:04 smithi079 ceph-mon[27222]: from='mgr.14100 172.21.15.79:0/3968367628' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "mon metadata", "id": "smithi079"}]: dispatch 2024-05-04T12:08:04.945 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:08:04 smithi079 ceph-mon[27222]: from='mgr.14100 172.21.15.79:0/3968367628' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "mgr metadata", "who": "smithi079.dgcpwl", "id": "smithi079.dgcpwl"}]: dispatch 2024-05-04T12:08:04.945 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:08:04 smithi079 ceph-mon[27222]: Manager daemon smithi079.dgcpwl is now available 2024-05-04T12:08:04.946 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:08:04 smithi079 ceph-mon[27222]: from='mgr.14100 172.21.15.79:0/3968367628' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi079.dgcpwl/mirror_snapshot_schedule"}]: dispatch 2024-05-04T12:08:04.946 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:08:04 smithi079 ceph-mon[27222]: from='mgr.14100 172.21.15.79:0/3968367628' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi079.dgcpwl/trash_purge_schedule"}]: dispatch 2024-05-04T12:08:04.946 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:08:04 smithi079 ceph-mon[27222]: from='mgr.14100 172.21.15.79:0/3968367628' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:08:04.946 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:08:04 smithi079 ceph-mon[27222]: from='mgr.14100 172.21.15.79:0/3968367628' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:08:04.946 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:08:04 smithi079 ceph-mon[27222]: from='mgr.14100 172.21.15.79:0/3968367628' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:08:05.272 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout 2024-05-04T12:08:05.272 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout { 2024-05-04T12:08:05.272 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "fsid": "d9b985c4-0a0e-11ef-bc97-c7b262605968", 2024-05-04T12:08:05.272 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "health": { 2024-05-04T12:08:05.272 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-05-04T12:08:05.273 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-05-04T12:08:05.273 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-05-04T12:08:05.273 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-05-04T12:08:05.273 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-05-04T12:08:05.273 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-05-04T12:08:05.273 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout 0 2024-05-04T12:08:05.273 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout ], 2024-05-04T12:08:05.273 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-05-04T12:08:05.273 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "smithi079" 2024-05-04T12:08:05.273 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout ], 2024-05-04T12:08:05.273 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "quorum_age": 8, 2024-05-04T12:08:05.273 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "monmap": { 2024-05-04T12:08:05.273 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-04T12:08:05.273 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-05-04T12:08:05.273 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-05-04T12:08:05.274 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-05-04T12:08:05.274 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-05-04T12:08:05.274 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-04T12:08:05.274 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-05-04T12:08:05.274 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-05-04T12:08:05.274 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-05-04T12:08:05.282 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-05-04T12:08:05.282 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-05-04T12:08:05.282 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-05-04T12:08:05.282 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-05-04T12:08:05.282 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-05-04T12:08:05.282 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-05-04T12:08:05.282 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-05-04T12:08:05.282 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-05-04T12:08:05.282 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-05-04T12:08:05.283 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-05-04T12:08:05.283 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-05-04T12:08:05.283 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-05-04T12:08:05.283 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-05-04T12:08:05.283 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-05-04T12:08:05.283 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-05-04T12:08:05.283 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-04T12:08:05.283 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-05-04T12:08:05.283 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-05-04T12:08:05.283 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-05-04T12:08:05.283 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-05-04T12:08:05.283 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "available": false, 2024-05-04T12:08:05.283 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-05-04T12:08:05.283 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "modules": [ 2024-05-04T12:08:05.284 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "iostat", 2024-05-04T12:08:05.284 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "nfs", 2024-05-04T12:08:05.284 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "restful" 2024-05-04T12:08:05.284 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout ], 2024-05-04T12:08:05.284 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-04T12:08:05.284 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-05-04T12:08:05.284 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-05-04T12:08:05.284 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-04T12:08:05.284 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "modified": "2024-05-04T12:07:51.285519+0000", 2024-05-04T12:08:05.284 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-04T12:08:05.284 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-05-04T12:08:05.284 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-05-04T12:08:05.284 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout } 2024-05-04T12:08:05.284 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:04.699+0000 7fc82d3bb700 1 Processor -- start 2024-05-04T12:08:05.285 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:04.699+0000 7fc82d3bb700 1 -- start start 2024-05-04T12:08:05.285 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:04.699+0000 7fc82d3bb700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc828105470 0x7fc828105890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:08:05.285 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:04.699+0000 7fc82d3bb700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc828105e60 con 0x7fc828105470 2024-05-04T12:08:05.285 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:04.700+0000 7fc826ffd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc828105470 0x7fc828105890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:08:05.285 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:04.700+0000 7fc826ffd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc828105470 0x7fc828105890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:42440/0 (socket says 172.21.15.79:42440) 2024-05-04T12:08:05.285 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:04.700+0000 7fc826ffd700 1 -- 172.21.15.79:0/2915032053 learned_addr learned my addr 172.21.15.79:0/2915032053 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:08:05.285 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:04.700+0000 7fc826ffd700 1 -- 172.21.15.79:0/2915032053 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc828106670 con 0x7fc828105470 2024-05-04T12:08:05.285 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:04.700+0000 7fc826ffd700 1 --2- 172.21.15.79:0/2915032053 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc828105470 0x7fc828105890 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fc8180098a0 tx=0x7fc818009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33314a530548accd server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:08:05.285 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:04.700+0000 7fc8267fc700 1 -- 172.21.15.79:0/2915032053 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc818004180 con 0x7fc828105470 2024-05-04T12:08:05.285 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:04.700+0000 7fc8267fc700 1 -- 172.21.15.79:0/2915032053 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fc8180042e0 con 0x7fc828105470 2024-05-04T12:08:05.285 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:04.700+0000 7fc8267fc700 1 -- 172.21.15.79:0/2915032053 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc818011a50 con 0x7fc828105470 2024-05-04T12:08:05.285 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:04.701+0000 7fc82d3bb700 1 -- 172.21.15.79:0/2915032053 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc828105470 msgr2=0x7fc828105890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:08:05.285 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:04.701+0000 7fc82d3bb700 1 --2- 172.21.15.79:0/2915032053 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc828105470 0x7fc828105890 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fc8180098a0 tx=0x7fc818009bb0 comp rx=0 tx=0).stop 2024-05-04T12:08:05.286 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:04.701+0000 7fc82d3bb700 1 -- 172.21.15.79:0/2915032053 shutdown_connections 2024-05-04T12:08:05.286 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:04.701+0000 7fc82d3bb700 1 --2- 172.21.15.79:0/2915032053 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc828105470 0x7fc828105890 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:08:05.286 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:04.701+0000 7fc82d3bb700 1 -- 172.21.15.79:0/2915032053 >> 172.21.15.79:0/2915032053 conn(0x7fc8281009f0 msgr2=0x7fc828102e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:08:05.286 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:04.701+0000 7fc82d3bb700 1 -- 172.21.15.79:0/2915032053 shutdown_connections 2024-05-04T12:08:05.286 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:04.701+0000 7fc82d3bb700 1 -- 172.21.15.79:0/2915032053 wait complete. 2024-05-04T12:08:05.286 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:04.701+0000 7fc82d3bb700 1 Processor -- start 2024-05-04T12:08:05.286 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:04.702+0000 7fc82d3bb700 1 -- start start 2024-05-04T12:08:05.286 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:04.702+0000 7fc82d3bb700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc828105470 0x7fc82819a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:08:05.286 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:04.702+0000 7fc82d3bb700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc82819a6f0 con 0x7fc828105470 2024-05-04T12:08:05.286 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:04.702+0000 7fc826ffd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc828105470 0x7fc82819a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:08:05.286 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:04.702+0000 7fc826ffd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc828105470 0x7fc82819a1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:42450/0 (socket says 172.21.15.79:42450) 2024-05-04T12:08:05.286 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:04.702+0000 7fc826ffd700 1 -- 172.21.15.79:0/749682702 learned_addr learned my addr 172.21.15.79:0/749682702 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:08:05.286 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:04.702+0000 7fc826ffd700 1 -- 172.21.15.79:0/749682702 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc818009550 con 0x7fc828105470 2024-05-04T12:08:05.286 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:04.702+0000 7fc826ffd700 1 --2- 172.21.15.79:0/749682702 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc828105470 0x7fc82819a1b0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fc818000c00 tx=0x7fc818011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:08:05.287 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:04.702+0000 7fc824ff9700 1 -- 172.21.15.79:0/749682702 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc818003b90 con 0x7fc828105470 2024-05-04T12:08:05.287 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:04.702+0000 7fc824ff9700 1 -- 172.21.15.79:0/749682702 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fc8180248b0 con 0x7fc828105470 2024-05-04T12:08:05.287 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:04.702+0000 7fc82d3bb700 1 -- 172.21.15.79:0/749682702 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc82819a8f0 con 0x7fc828105470 2024-05-04T12:08:05.287 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:04.702+0000 7fc824ff9700 1 -- 172.21.15.79:0/749682702 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc81801a8e0 con 0x7fc828105470 2024-05-04T12:08:05.287 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:04.702+0000 7fc82d3bb700 1 -- 172.21.15.79:0/749682702 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc82819ad90 con 0x7fc828105470 2024-05-04T12:08:05.287 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:04.703+0000 7fc824ff9700 1 -- 172.21.15.79:0/749682702 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 2) v1 ==== 45039+0+0 (secure 0 0 0) 0x7fc81801aba0 con 0x7fc828105470 2024-05-04T12:08:05.287 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:04.703+0000 7fc82d3bb700 1 -- 172.21.15.79:0/749682702 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc808005320 con 0x7fc828105470 2024-05-04T12:08:05.287 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:04.704+0000 7fc824ff9700 1 -- 172.21.15.79:0/749682702 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fc81804bde0 con 0x7fc828105470 2024-05-04T12:08:05.287 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:04.706+0000 7fc824ff9700 1 -- 172.21.15.79:0/749682702 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7fc818018960 con 0x7fc828105470 2024-05-04T12:08:05.287 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:04.774+0000 7fc82d3bb700 1 -- 172.21.15.79:0/749682702 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fc808005190 con 0x7fc828105470 2024-05-04T12:08:05.287 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:04.774+0000 7fc824ff9700 1 -- 172.21.15.79:0/749682702 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1246 (secure 0 0 0) 0x7fc818018960 con 0x7fc828105470 2024-05-04T12:08:05.287 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:04.775+0000 7fc82d3bb700 1 -- 172.21.15.79:0/749682702 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc828105470 msgr2=0x7fc82819a1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:08:05.287 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:04.775+0000 7fc82d3bb700 1 --2- 172.21.15.79:0/749682702 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc828105470 0x7fc82819a1b0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fc818000c00 tx=0x7fc818011ab0 comp rx=0 tx=0).stop 2024-05-04T12:08:05.287 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:04.775+0000 7fc82d3bb700 1 -- 172.21.15.79:0/749682702 shutdown_connections 2024-05-04T12:08:05.287 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:04.775+0000 7fc82d3bb700 1 --2- 172.21.15.79:0/749682702 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc828105470 0x7fc82819a1b0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:08:05.287 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:04.775+0000 7fc82d3bb700 1 -- 172.21.15.79:0/749682702 >> 172.21.15.79:0/749682702 conn(0x7fc8281009f0 msgr2=0x7fc828101740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:08:05.288 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:04.775+0000 7fc82d3bb700 1 -- 172.21.15.79:0/749682702 shutdown_connections 2024-05-04T12:08:05.288 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:04.775+0000 7fc82d3bb700 1 -- 172.21.15.79:0/749682702 wait complete. 2024-05-04T12:08:05.288 INFO:teuthology.orchestra.run.smithi079.stdout:mgr not available, waiting (3/15)... 2024-05-04T12:08:05.945 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:08:05 smithi079 ceph-mon[27222]: from='client.? 172.21.15.79:0/749682702' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-05-04T12:08:05.945 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:08:05 smithi079 ceph-mon[27222]: mgrmap e3: smithi079.dgcpwl(active, since 1.00648s) 2024-05-04T12:08:08.304 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:08:07 smithi079 ceph-mon[27222]: mgrmap e4: smithi079.dgcpwl(active, since 2s) 2024-05-04T12:08:10.944 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:08:10 smithi079 ceph-mon[27222]: from='client.? 172.21.15.79:0/2580492503' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-05-04T12:08:16.953 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout 2024-05-04T12:08:16.956 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout { 2024-05-04T12:08:16.956 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "fsid": "d9b985c4-0a0e-11ef-bc97-c7b262605968", 2024-05-04T12:08:16.956 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "health": { 2024-05-04T12:08:16.956 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-05-04T12:08:16.956 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-05-04T12:08:16.956 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-05-04T12:08:16.957 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-05-04T12:08:16.957 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-05-04T12:08:16.957 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-05-04T12:08:16.957 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout 0 2024-05-04T12:08:16.957 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout ], 2024-05-04T12:08:16.957 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-05-04T12:08:16.957 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "smithi079" 2024-05-04T12:08:16.957 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout ], 2024-05-04T12:08:16.957 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "quorum_age": 14, 2024-05-04T12:08:16.957 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "monmap": { 2024-05-04T12:08:16.957 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-04T12:08:16.957 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-05-04T12:08:16.957 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-05-04T12:08:16.957 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-05-04T12:08:16.957 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-05-04T12:08:16.958 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-04T12:08:16.958 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-05-04T12:08:16.958 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-05-04T12:08:16.958 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-05-04T12:08:16.966 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-05-04T12:08:16.966 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-05-04T12:08:16.966 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-05-04T12:08:16.966 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-05-04T12:08:16.966 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-05-04T12:08:16.966 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-05-04T12:08:16.967 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-05-04T12:08:16.967 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-05-04T12:08:16.967 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-05-04T12:08:16.967 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-05-04T12:08:16.967 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-05-04T12:08:16.967 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-05-04T12:08:16.967 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-05-04T12:08:16.967 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-05-04T12:08:16.967 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-05-04T12:08:16.967 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-04T12:08:16.967 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-05-04T12:08:16.967 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-05-04T12:08:16.967 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-05-04T12:08:16.968 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-05-04T12:08:16.968 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "available": true, 2024-05-04T12:08:16.968 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-05-04T12:08:16.968 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "modules": [ 2024-05-04T12:08:16.968 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "iostat", 2024-05-04T12:08:16.968 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "nfs", 2024-05-04T12:08:16.968 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "restful" 2024-05-04T12:08:16.968 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout ], 2024-05-04T12:08:16.968 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-04T12:08:16.968 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-05-04T12:08:16.968 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-05-04T12:08:16.968 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-04T12:08:16.969 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "modified": "2024-05-04T12:07:51.285519+0000", 2024-05-04T12:08:16.969 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-04T12:08:16.969 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-05-04T12:08:16.969 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-05-04T12:08:16.969 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout } 2024-05-04T12:08:16.969 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:10.204+0000 7fb24a76b700 1 Processor -- start 2024-05-04T12:08:16.969 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:10.204+0000 7fb24a76b700 1 -- start start 2024-05-04T12:08:16.969 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:10.204+0000 7fb24a76b700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb24407bea0 0x7fb24407a300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:08:16.969 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:10.204+0000 7fb24a76b700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb24407a8d0 con 0x7fb24407bea0 2024-05-04T12:08:16.969 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:10.204+0000 7fb243fff700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb24407bea0 0x7fb24407a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:08:16.969 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:10.204+0000 7fb243fff700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb24407bea0 0x7fb24407a300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:42466/0 (socket says 172.21.15.79:42466) 2024-05-04T12:08:16.969 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:10.204+0000 7fb243fff700 1 -- 172.21.15.79:0/45514244 learned_addr learned my addr 172.21.15.79:0/45514244 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:08:16.969 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:10.205+0000 7fb243fff700 1 -- 172.21.15.79:0/45514244 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb24407aa10 con 0x7fb24407bea0 2024-05-04T12:08:16.969 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:10.205+0000 7fb243fff700 1 --2- 172.21.15.79:0/45514244 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb24407bea0 0x7fb24407a300 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fb2340098a0 tx=0x7fb234009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47f1bb437eb4fc68 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:08:16.970 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:10.205+0000 7fb242ffd700 1 -- 172.21.15.79:0/45514244 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb234004180 con 0x7fb24407bea0 2024-05-04T12:08:16.970 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:10.205+0000 7fb242ffd700 1 -- 172.21.15.79:0/45514244 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fb2340042e0 con 0x7fb24407bea0 2024-05-04T12:08:16.970 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:10.205+0000 7fb242ffd700 1 -- 172.21.15.79:0/45514244 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb234011a50 con 0x7fb24407bea0 2024-05-04T12:08:16.970 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:10.206+0000 7fb24a76b700 1 -- 172.21.15.79:0/45514244 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb24407bea0 msgr2=0x7fb24407a300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:08:16.970 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:10.206+0000 7fb24a76b700 1 --2- 172.21.15.79:0/45514244 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb24407bea0 0x7fb24407a300 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fb2340098a0 tx=0x7fb234009bb0 comp rx=0 tx=0).stop 2024-05-04T12:08:16.970 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:10.206+0000 7fb24a76b700 1 -- 172.21.15.79:0/45514244 shutdown_connections 2024-05-04T12:08:16.970 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:10.206+0000 7fb24a76b700 1 --2- 172.21.15.79:0/45514244 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb24407bea0 0x7fb24407a300 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:08:16.970 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:10.206+0000 7fb24a76b700 1 -- 172.21.15.79:0/45514244 >> 172.21.15.79:0/45514244 conn(0x7fb244101b40 msgr2=0x7fb244103f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:08:16.970 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:10.206+0000 7fb24a76b700 1 -- 172.21.15.79:0/45514244 shutdown_connections 2024-05-04T12:08:16.970 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:10.206+0000 7fb24a76b700 1 -- 172.21.15.79:0/45514244 wait complete. 2024-05-04T12:08:16.970 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:10.206+0000 7fb24a76b700 1 Processor -- start 2024-05-04T12:08:16.970 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:10.206+0000 7fb24a76b700 1 -- start start 2024-05-04T12:08:16.970 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:10.207+0000 7fb24a76b700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb24407bea0 0x7fb24419c420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:08:16.970 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:10.207+0000 7fb24a76b700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb24419c960 con 0x7fb24407bea0 2024-05-04T12:08:16.971 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:10.207+0000 7fb243fff700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb24407bea0 0x7fb24419c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:08:16.971 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:10.207+0000 7fb243fff700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb24407bea0 0x7fb24419c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:42480/0 (socket says 172.21.15.79:42480) 2024-05-04T12:08:16.971 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:10.207+0000 7fb243fff700 1 -- 172.21.15.79:0/2580492503 learned_addr learned my addr 172.21.15.79:0/2580492503 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:08:16.971 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:10.207+0000 7fb243fff700 1 -- 172.21.15.79:0/2580492503 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb234009550 con 0x7fb24407bea0 2024-05-04T12:08:16.971 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:10.207+0000 7fb243fff700 1 --2- 172.21.15.79:0/2580492503 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb24407bea0 0x7fb24419c420 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fb234003de0 tx=0x7fb234011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:08:16.971 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:10.207+0000 7fb2417fa700 1 -- 172.21.15.79:0/2580492503 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb234003930 con 0x7fb24407bea0 2024-05-04T12:08:16.971 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:10.207+0000 7fb24a76b700 1 -- 172.21.15.79:0/2580492503 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb24419cb60 con 0x7fb24407bea0 2024-05-04T12:08:16.971 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:10.207+0000 7fb2417fa700 1 -- 172.21.15.79:0/2580492503 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fb234003a90 con 0x7fb24407bea0 2024-05-04T12:08:16.971 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:10.207+0000 7fb2417fa700 1 -- 172.21.15.79:0/2580492503 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb2340196c0 con 0x7fb24407bea0 2024-05-04T12:08:16.971 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:10.207+0000 7fb24a76b700 1 -- 172.21.15.79:0/2580492503 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb24419d000 con 0x7fb24407bea0 2024-05-04T12:08:16.971 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:10.208+0000 7fb2417fa700 1 -- 172.21.15.79:0/2580492503 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 4) v1 ==== 45272+0+0 (secure 0 0 0) 0x7fb234019980 con 0x7fb24407bea0 2024-05-04T12:08:16.971 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:10.208+0000 7fb24a76b700 1 -- 172.21.15.79:0/2580492503 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb24410c8a0 con 0x7fb24407bea0 2024-05-04T12:08:16.971 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:10.208+0000 7fb2417fa700 1 --2- 172.21.15.79:0/2580492503 >> [v2:172.21.15.79:6800/839275478,v1:172.21.15.79:6801/839275478] conn(0x7fb22c0382d0 0x7fb22c03a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:08:16.971 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:10.208+0000 7fb2417fa700 1 -- 172.21.15.79:0/2580492503 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fb23404c140 con 0x7fb24407bea0 2024-05-04T12:08:16.971 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:10.209+0000 7fb2437fe700 1 --2- 172.21.15.79:0/2580492503 >> [v2:172.21.15.79:6800/839275478,v1:172.21.15.79:6801/839275478] conn(0x7fb22c0382d0 0x7fb22c03a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:08:16.972 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:10.209+0000 7fb2437fe700 1 --2- 172.21.15.79:0/2580492503 >> [v2:172.21.15.79:6800/839275478,v1:172.21.15.79:6801/839275478] conn(0x7fb22c0382d0 0x7fb22c03a790 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fb238006fd0 tx=0x7fb238006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:08:16.972 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:10.213+0000 7fb2417fa700 1 -- 172.21.15.79:0/2580492503 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fb234019c30 con 0x7fb24407bea0 2024-05-04T12:08:16.972 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:10.487+0000 7fb24a76b700 1 -- 172.21.15.79:0/2580492503 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fb24410cac0 con 0x7fb24407bea0 2024-05-04T12:08:16.972 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:10.488+0000 7fb2417fa700 1 -- 172.21.15.79:0/2580492503 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1246 (secure 0 0 0) 0x7fb23401a770 con 0x7fb24407bea0 2024-05-04T12:08:16.972 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:10.490+0000 7fb24a76b700 1 -- 172.21.15.79:0/2580492503 >> [v2:172.21.15.79:6800/839275478,v1:172.21.15.79:6801/839275478] conn(0x7fb22c0382d0 msgr2=0x7fb22c03a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:08:16.972 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:10.490+0000 7fb24a76b700 1 --2- 172.21.15.79:0/2580492503 >> [v2:172.21.15.79:6800/839275478,v1:172.21.15.79:6801/839275478] conn(0x7fb22c0382d0 0x7fb22c03a790 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fb238006fd0 tx=0x7fb238006e40 comp rx=0 tx=0).stop 2024-05-04T12:08:16.972 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:10.490+0000 7fb24a76b700 1 -- 172.21.15.79:0/2580492503 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb24407bea0 msgr2=0x7fb24419c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:08:16.972 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:10.490+0000 7fb24a76b700 1 --2- 172.21.15.79:0/2580492503 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb24407bea0 0x7fb24419c420 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fb234003de0 tx=0x7fb234011ab0 comp rx=0 tx=0).stop 2024-05-04T12:08:16.972 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:10.490+0000 7fb24a76b700 1 -- 172.21.15.79:0/2580492503 shutdown_connections 2024-05-04T12:08:16.972 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:10.490+0000 7fb24a76b700 1 --2- 172.21.15.79:0/2580492503 >> [v2:172.21.15.79:6800/839275478,v1:172.21.15.79:6801/839275478] conn(0x7fb22c0382d0 0x7fb22c03a790 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:08:16.972 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:10.490+0000 7fb24a76b700 1 --2- 172.21.15.79:0/2580492503 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb24407bea0 0x7fb24419c420 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:08:16.972 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:10.490+0000 7fb24a76b700 1 -- 172.21.15.79:0/2580492503 >> 172.21.15.79:0/2580492503 conn(0x7fb244101b40 msgr2=0x7fb244103190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:08:16.972 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:10.490+0000 7fb24a76b700 1 -- 172.21.15.79:0/2580492503 shutdown_connections 2024-05-04T12:08:16.972 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:10.490+0000 7fb24a76b700 1 -- 172.21.15.79:0/2580492503 wait complete. 2024-05-04T12:08:16.972 INFO:teuthology.orchestra.run.smithi079.stdout:mgr is available 2024-05-04T12:08:18.695 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:08:18 smithi079 ceph-mon[27222]: from='client.? 172.21.15.79:0/3286368594' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-05-04T12:08:18.769 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout 2024-05-04T12:08:18.769 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout [global] 2024-05-04T12:08:18.769 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout fsid = d9b985c4-0a0e-11ef-bc97-c7b262605968 2024-05-04T12:08:18.769 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-05-04T12:08:18.769 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-05-04T12:08:18.769 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-05-04T12:08:18.769 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-05-04T12:08:18.770 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout 2024-05-04T12:08:18.770 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout [mgr] 2024-05-04T12:08:18.770 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-05-04T12:08:18.770 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout 2024-05-04T12:08:18.770 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout [osd] 2024-05-04T12:08:18.770 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-05-04T12:08:18.770 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-05-04T12:08:18.770 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-05-04T12:08:18.771 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-05-04T12:08:18.771 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-05-04T12:08:18.771 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:17.999+0000 7fba8aeb2700 1 Processor -- start 2024-05-04T12:08:18.771 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:17.999+0000 7fba8aeb2700 1 -- start start 2024-05-04T12:08:18.771 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:17.999+0000 7fba8aeb2700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fba841070c0 0x7fba841094b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:08:18.771 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:17.999+0000 7fba8aeb2700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba840745a0 con 0x7fba841070c0 2024-05-04T12:08:18.771 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:18.000+0000 7fba88c4e700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fba841070c0 0x7fba841094b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:08:18.772 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:18.000+0000 7fba88c4e700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fba841070c0 0x7fba841094b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:49258/0 (socket says 172.21.15.79:49258) 2024-05-04T12:08:18.772 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:18.000+0000 7fba88c4e700 1 -- 172.21.15.79:0/3697099536 learned_addr learned my addr 172.21.15.79:0/3697099536 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:08:18.772 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:18.000+0000 7fba88c4e700 1 -- 172.21.15.79:0/3697099536 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba840746e0 con 0x7fba841070c0 2024-05-04T12:08:18.772 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:18.000+0000 7fba88c4e700 1 --2- 172.21.15.79:0/3697099536 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fba841070c0 0x7fba841094b0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fba780098a0 tx=0x7fba78009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62aa19a1904abaa2 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:08:18.772 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:18.000+0000 7fba837fe700 1 -- 172.21.15.79:0/3697099536 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fba78004180 con 0x7fba841070c0 2024-05-04T12:08:18.772 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:18.000+0000 7fba837fe700 1 -- 172.21.15.79:0/3697099536 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fba780042e0 con 0x7fba841070c0 2024-05-04T12:08:18.772 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:18.000+0000 7fba837fe700 1 -- 172.21.15.79:0/3697099536 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fba78011a50 con 0x7fba841070c0 2024-05-04T12:08:18.773 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:18.001+0000 7fba8aeb2700 1 -- 172.21.15.79:0/3697099536 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fba841070c0 msgr2=0x7fba841094b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:08:18.773 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:18.001+0000 7fba8aeb2700 1 --2- 172.21.15.79:0/3697099536 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fba841070c0 0x7fba841094b0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fba780098a0 tx=0x7fba78009bb0 comp rx=0 tx=0).stop 2024-05-04T12:08:18.773 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:18.001+0000 7fba8aeb2700 1 -- 172.21.15.79:0/3697099536 shutdown_connections 2024-05-04T12:08:18.773 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:18.001+0000 7fba8aeb2700 1 --2- 172.21.15.79:0/3697099536 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fba841070c0 0x7fba841094b0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:08:18.773 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:18.001+0000 7fba8aeb2700 1 -- 172.21.15.79:0/3697099536 >> 172.21.15.79:0/3697099536 conn(0x7fba841009f0 msgr2=0x7fba84102e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:08:18.773 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:18.001+0000 7fba8aeb2700 1 -- 172.21.15.79:0/3697099536 shutdown_connections 2024-05-04T12:08:18.774 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:18.001+0000 7fba8aeb2700 1 -- 172.21.15.79:0/3697099536 wait complete. 2024-05-04T12:08:18.774 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:18.002+0000 7fba8aeb2700 1 Processor -- start 2024-05-04T12:08:18.774 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:18.002+0000 7fba8aeb2700 1 -- start start 2024-05-04T12:08:18.774 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:18.002+0000 7fba8aeb2700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fba841070c0 0x7fba84106a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:08:18.774 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:18.002+0000 7fba8aeb2700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba84106f60 con 0x7fba841070c0 2024-05-04T12:08:18.774 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:18.002+0000 7fba88c4e700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fba841070c0 0x7fba84106a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:08:18.775 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:18.002+0000 7fba88c4e700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fba841070c0 0x7fba84106a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:49264/0 (socket says 172.21.15.79:49264) 2024-05-04T12:08:18.775 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:18.002+0000 7fba88c4e700 1 -- 172.21.15.79:0/3286368594 learned_addr learned my addr 172.21.15.79:0/3286368594 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:08:18.775 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:18.002+0000 7fba88c4e700 1 -- 172.21.15.79:0/3286368594 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba78009550 con 0x7fba841070c0 2024-05-04T12:08:18.775 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:18.002+0000 7fba88c4e700 1 --2- 172.21.15.79:0/3286368594 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fba841070c0 0x7fba84106a20 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fba78000c00 tx=0x7fba78011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:08:18.775 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:18.002+0000 7fba81ffb700 1 -- 172.21.15.79:0/3286368594 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fba78003b90 con 0x7fba841070c0 2024-05-04T12:08:18.775 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:18.003+0000 7fba81ffb700 1 -- 172.21.15.79:0/3286368594 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fba780248b0 con 0x7fba841070c0 2024-05-04T12:08:18.775 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:18.003+0000 7fba8aeb2700 1 -- 172.21.15.79:0/3286368594 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba84105070 con 0x7fba841070c0 2024-05-04T12:08:18.775 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:18.003+0000 7fba81ffb700 1 -- 172.21.15.79:0/3286368594 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fba7801a8e0 con 0x7fba841070c0 2024-05-04T12:08:18.775 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:18.003+0000 7fba8aeb2700 1 -- 172.21.15.79:0/3286368594 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba84105510 con 0x7fba841070c0 2024-05-04T12:08:18.775 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:18.004+0000 7fba81ffb700 1 -- 172.21.15.79:0/3286368594 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 4) v1 ==== 45272+0+0 (secure 0 0 0) 0x7fba7801aba0 con 0x7fba841070c0 2024-05-04T12:08:18.776 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:18.004+0000 7fba81ffb700 1 --2- 172.21.15.79:0/3286368594 >> [v2:172.21.15.79:6800/839275478,v1:172.21.15.79:6801/839275478] conn(0x7fba740382d0 0x7fba7403a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:08:18.776 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:18.004+0000 7fba8aeb2700 1 -- 172.21.15.79:0/3286368594 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba8410a6a0 con 0x7fba841070c0 2024-05-04T12:08:18.776 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:18.004+0000 7fba81ffb700 1 -- 172.21.15.79:0/3286368594 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fba7804c1d0 con 0x7fba841070c0 2024-05-04T12:08:18.776 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:18.004+0000 7fba83fff700 1 --2- 172.21.15.79:0/3286368594 >> [v2:172.21.15.79:6800/839275478,v1:172.21.15.79:6801/839275478] conn(0x7fba740382d0 0x7fba7403a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:08:18.776 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:18.004+0000 7fba83fff700 1 --2- 172.21.15.79:0/3286368594 >> [v2:172.21.15.79:6800/839275478,v1:172.21.15.79:6801/839275478] conn(0x7fba740382d0 0x7fba7403a790 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fba70006fd0 tx=0x7fba70006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:08:18.776 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:18.009+0000 7fba81ffb700 1 -- 172.21.15.79:0/3286368594 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fba78018350 con 0x7fba841070c0 2024-05-04T12:08:18.776 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:18.195+0000 7fba8aeb2700 1 -- 172.21.15.79:0/3286368594 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7fba84105db0 con 0x7fba841070c0 2024-05-04T12:08:18.776 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:18.196+0000 7fba81ffb700 1 -- 172.21.15.79:0/3286368594 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+373 (secure 0 0 0) 0x7fba78018350 con 0x7fba841070c0 2024-05-04T12:08:18.776 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:18.198+0000 7fba8aeb2700 1 -- 172.21.15.79:0/3286368594 >> [v2:172.21.15.79:6800/839275478,v1:172.21.15.79:6801/839275478] conn(0x7fba740382d0 msgr2=0x7fba7403a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:08:18.776 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:18.198+0000 7fba8aeb2700 1 --2- 172.21.15.79:0/3286368594 >> [v2:172.21.15.79:6800/839275478,v1:172.21.15.79:6801/839275478] conn(0x7fba740382d0 0x7fba7403a790 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fba70006fd0 tx=0x7fba70006e40 comp rx=0 tx=0).stop 2024-05-04T12:08:18.776 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:18.198+0000 7fba8aeb2700 1 -- 172.21.15.79:0/3286368594 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fba841070c0 msgr2=0x7fba84106a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:08:18.776 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:18.198+0000 7fba8aeb2700 1 --2- 172.21.15.79:0/3286368594 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fba841070c0 0x7fba84106a20 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fba78000c00 tx=0x7fba78011ab0 comp rx=0 tx=0).stop 2024-05-04T12:08:18.776 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:18.199+0000 7fba8aeb2700 1 -- 172.21.15.79:0/3286368594 shutdown_connections 2024-05-04T12:08:18.777 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:18.199+0000 7fba8aeb2700 1 --2- 172.21.15.79:0/3286368594 >> [v2:172.21.15.79:6800/839275478,v1:172.21.15.79:6801/839275478] conn(0x7fba740382d0 0x7fba7403a790 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:08:18.777 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:18.199+0000 7fba8aeb2700 1 --2- 172.21.15.79:0/3286368594 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fba841070c0 0x7fba84106a20 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:08:18.777 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:18.199+0000 7fba8aeb2700 1 -- 172.21.15.79:0/3286368594 >> 172.21.15.79:0/3286368594 conn(0x7fba841009f0 msgr2=0x7fba84101740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:08:18.777 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:18.199+0000 7fba8aeb2700 1 -- 172.21.15.79:0/3286368594 shutdown_connections 2024-05-04T12:08:18.777 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:18.199+0000 7fba8aeb2700 1 -- 172.21.15.79:0/3286368594 wait complete. 2024-05-04T12:08:18.777 INFO:teuthology.orchestra.run.smithi079.stdout:Enabling cephadm module... 2024-05-04T12:08:19.807 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:08:19 smithi079 ceph-mon[27222]: from='client.? 172.21.15.79:0/3410248397' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-05-04T12:08:20.083 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:19.211+0000 7f901b993700 1 Processor -- start 2024-05-04T12:08:20.084 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:19.211+0000 7f901b993700 1 -- start start 2024-05-04T12:08:20.084 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:19.211+0000 7f901b993700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f901407be40 0x7f901407a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:08:20.084 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:19.211+0000 7f901b993700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f901407a870 con 0x7f901407be40 2024-05-04T12:08:20.084 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:19.211+0000 7f901972f700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f901407be40 0x7f901407a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:08:20.084 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:19.211+0000 7f901972f700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f901407be40 0x7f901407a2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:49274/0 (socket says 172.21.15.79:49274) 2024-05-04T12:08:20.084 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:19.211+0000 7f901972f700 1 -- 172.21.15.79:0/158805397 learned_addr learned my addr 172.21.15.79:0/158805397 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:08:20.084 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:19.211+0000 7f901972f700 1 -- 172.21.15.79:0/158805397 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f901407a9b0 con 0x7f901407be40 2024-05-04T12:08:20.084 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:19.212+0000 7f901972f700 1 --2- 172.21.15.79:0/158805397 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f901407be40 0x7f901407a2a0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f90100098a0 tx=0x7f9010009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0f673ad04725dd7 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:08:20.084 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:19.212+0000 7f900bfff700 1 -- 172.21.15.79:0/158805397 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9010004180 con 0x7f901407be40 2024-05-04T12:08:20.084 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:19.212+0000 7f900bfff700 1 -- 172.21.15.79:0/158805397 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f90100042e0 con 0x7f901407be40 2024-05-04T12:08:20.084 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:19.212+0000 7f900bfff700 1 -- 172.21.15.79:0/158805397 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9010011a50 con 0x7f901407be40 2024-05-04T12:08:20.084 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:19.212+0000 7f901b993700 1 -- 172.21.15.79:0/158805397 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f901407be40 msgr2=0x7f901407a2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:08:20.084 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:19.212+0000 7f901b993700 1 --2- 172.21.15.79:0/158805397 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f901407be40 0x7f901407a2a0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f90100098a0 tx=0x7f9010009bb0 comp rx=0 tx=0).stop 2024-05-04T12:08:20.085 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:19.212+0000 7f901b993700 1 -- 172.21.15.79:0/158805397 shutdown_connections 2024-05-04T12:08:20.085 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:19.212+0000 7f901b993700 1 --2- 172.21.15.79:0/158805397 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f901407be40 0x7f901407a2a0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:08:20.085 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:19.212+0000 7f901b993700 1 -- 172.21.15.79:0/158805397 >> 172.21.15.79:0/158805397 conn(0x7f9014101b40 msgr2=0x7f9014103f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:08:20.085 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:19.212+0000 7f901b993700 1 -- 172.21.15.79:0/158805397 shutdown_connections 2024-05-04T12:08:20.085 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:19.212+0000 7f901b993700 1 -- 172.21.15.79:0/158805397 wait complete. 2024-05-04T12:08:20.085 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:19.213+0000 7f901b993700 1 Processor -- start 2024-05-04T12:08:20.085 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:19.213+0000 7f901b993700 1 -- start start 2024-05-04T12:08:20.085 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:19.213+0000 7f901b993700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f901407be40 0x7f901419c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:08:20.085 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:19.213+0000 7f901b993700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f901419c970 con 0x7f901407be40 2024-05-04T12:08:20.085 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:19.213+0000 7f901972f700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f901407be40 0x7f901419c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:08:20.085 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:19.213+0000 7f901972f700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f901407be40 0x7f901419c430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:49276/0 (socket says 172.21.15.79:49276) 2024-05-04T12:08:20.086 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:19.213+0000 7f901972f700 1 -- 172.21.15.79:0/3410248397 learned_addr learned my addr 172.21.15.79:0/3410248397 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:08:20.086 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:19.214+0000 7f901972f700 1 -- 172.21.15.79:0/3410248397 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9010009550 con 0x7f901407be40 2024-05-04T12:08:20.086 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:19.214+0000 7f901972f700 1 --2- 172.21.15.79:0/3410248397 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f901407be40 0x7f901419c430 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f9010003de0 tx=0x7f9010011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:08:20.086 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:19.214+0000 7f900a7fc700 1 -- 172.21.15.79:0/3410248397 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9010003930 con 0x7f901407be40 2024-05-04T12:08:20.086 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:19.214+0000 7f900a7fc700 1 -- 172.21.15.79:0/3410248397 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f9010003a90 con 0x7f901407be40 2024-05-04T12:08:20.086 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:19.214+0000 7f901b993700 1 -- 172.21.15.79:0/3410248397 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f901419cb70 con 0x7f901407be40 2024-05-04T12:08:20.086 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:19.214+0000 7f900a7fc700 1 -- 172.21.15.79:0/3410248397 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9010019730 con 0x7f901407be40 2024-05-04T12:08:20.086 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:19.214+0000 7f901b993700 1 -- 172.21.15.79:0/3410248397 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f901419d010 con 0x7f901407be40 2024-05-04T12:08:20.086 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:19.215+0000 7f900a7fc700 1 -- 172.21.15.79:0/3410248397 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 4) v1 ==== 45272+0+0 (secure 0 0 0) 0x7f90100199b0 con 0x7f901407be40 2024-05-04T12:08:20.086 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:19.215+0000 7f901b993700 1 -- 172.21.15.79:0/3410248397 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f901410c880 con 0x7f901407be40 2024-05-04T12:08:20.086 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:19.215+0000 7f900a7fc700 1 --2- 172.21.15.79:0/3410248397 >> [v2:172.21.15.79:6800/839275478,v1:172.21.15.79:6801/839275478] conn(0x7f90000382d0 0x7f900003a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:08:20.086 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:19.215+0000 7f900a7fc700 1 -- 172.21.15.79:0/3410248397 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f901004c0c0 con 0x7f901407be40 2024-05-04T12:08:20.087 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:19.215+0000 7f9018f2e700 1 --2- 172.21.15.79:0/3410248397 >> [v2:172.21.15.79:6800/839275478,v1:172.21.15.79:6801/839275478] conn(0x7f90000382d0 0x7f900003a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:08:20.087 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:19.215+0000 7f9018f2e700 1 --2- 172.21.15.79:0/3410248397 >> [v2:172.21.15.79:6800/839275478,v1:172.21.15.79:6801/839275478] conn(0x7f90000382d0 0x7f900003a790 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f9004006fd0 tx=0x7f9004006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:08:20.087 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:19.220+0000 7f900a7fc700 1 -- 172.21.15.79:0/3410248397 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f901001abc0 con 0x7f901407be40 2024-05-04T12:08:20.087 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:19.460+0000 7f901b993700 1 -- 172.21.15.79:0/3410248397 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7f901419fdf0 con 0x7f901407be40 2024-05-04T12:08:20.087 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:19.512+0000 7f900a7fc700 1 -- 172.21.15.79:0/3410248397 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mgrmap(e 5) v1 ==== 45283+0+0 (secure 0 0 0) 0x7f901001a430 con 0x7f901407be40 2024-05-04T12:08:20.087 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:19.514+0000 7f900a7fc700 1 -- 172.21.15.79:0/3410248397 <== mon.0 v2:172.21.15.79:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7f90100281c0 con 0x7f901407be40 2024-05-04T12:08:20.087 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:19.517+0000 7f901b993700 1 -- 172.21.15.79:0/3410248397 >> [v2:172.21.15.79:6800/839275478,v1:172.21.15.79:6801/839275478] conn(0x7f90000382d0 msgr2=0x7f900003a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:08:20.087 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:19.517+0000 7f901b993700 1 --2- 172.21.15.79:0/3410248397 >> [v2:172.21.15.79:6800/839275478,v1:172.21.15.79:6801/839275478] conn(0x7f90000382d0 0x7f900003a790 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f9004006fd0 tx=0x7f9004006e40 comp rx=0 tx=0).stop 2024-05-04T12:08:20.087 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:19.517+0000 7f901b993700 1 -- 172.21.15.79:0/3410248397 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f901407be40 msgr2=0x7f901419c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:08:20.087 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:19.517+0000 7f901b993700 1 --2- 172.21.15.79:0/3410248397 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f901407be40 0x7f901419c430 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f9010003de0 tx=0x7f9010011ab0 comp rx=0 tx=0).stop 2024-05-04T12:08:20.087 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:19.517+0000 7f901b993700 1 -- 172.21.15.79:0/3410248397 shutdown_connections 2024-05-04T12:08:20.087 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:19.517+0000 7f901b993700 1 --2- 172.21.15.79:0/3410248397 >> [v2:172.21.15.79:6800/839275478,v1:172.21.15.79:6801/839275478] conn(0x7f90000382d0 0x7f900003a790 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:08:20.087 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:19.517+0000 7f901b993700 1 --2- 172.21.15.79:0/3410248397 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f901407be40 0x7f901419c430 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:08:20.087 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:19.517+0000 7f901b993700 1 -- 172.21.15.79:0/3410248397 >> 172.21.15.79:0/3410248397 conn(0x7f9014101b40 msgr2=0x7f9014103280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:08:20.088 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:19.517+0000 7f901b993700 1 -- 172.21.15.79:0/3410248397 shutdown_connections 2024-05-04T12:08:20.088 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:19.517+0000 7f901b993700 1 -- 172.21.15.79:0/3410248397 wait complete. 2024-05-04T12:08:20.695 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:08:20 smithi079 ceph-mon[27222]: from='client.? 172.21.15.79:0/3410248397' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-05-04T12:08:20.695 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:08:20 smithi079 ceph-mon[27222]: mgrmap e5: smithi079.dgcpwl(active, since 14s) 2024-05-04T12:08:21.331 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout { 2024-05-04T12:08:21.332 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-05-04T12:08:21.332 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "available": true, 2024-05-04T12:08:21.332 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "active_name": "smithi079.dgcpwl", 2024-05-04T12:08:21.332 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-05-04T12:08:21.332 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout } 2024-05-04T12:08:21.332 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:20.520+0000 7f2579bcb700 1 Processor -- start 2024-05-04T12:08:21.332 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:20.520+0000 7f2579bcb700 1 -- start start 2024-05-04T12:08:21.332 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:20.520+0000 7f2579bcb700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2574105970 0x7f2574105d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:08:21.332 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:20.520+0000 7f2579bcb700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2574106360 con 0x7f2574105970 2024-05-04T12:08:21.332 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:20.520+0000 7f25737fe700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2574105970 0x7f2574105d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:08:21.332 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:20.520+0000 7f25737fe700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2574105970 0x7f2574105d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:49300/0 (socket says 172.21.15.79:49300) 2024-05-04T12:08:21.332 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:20.520+0000 7f25737fe700 1 -- 172.21.15.79:0/3829237662 learned_addr learned my addr 172.21.15.79:0/3829237662 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:08:21.332 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:20.521+0000 7f25737fe700 1 -- 172.21.15.79:0/3829237662 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2574106690 con 0x7f2574105970 2024-05-04T12:08:21.333 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:20.521+0000 7f25737fe700 1 --2- 172.21.15.79:0/3829237662 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2574105970 0x7f2574105d90 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f25640098a0 tx=0x7f2564009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7695353fa162a4bc server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:08:21.333 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:20.521+0000 7f25727fc700 1 -- 172.21.15.79:0/3829237662 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2564004180 con 0x7f2574105970 2024-05-04T12:08:21.333 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:20.521+0000 7f25727fc700 1 -- 172.21.15.79:0/3829237662 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f25640042e0 con 0x7f2574105970 2024-05-04T12:08:21.333 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:20.521+0000 7f25727fc700 1 -- 172.21.15.79:0/3829237662 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2564011a50 con 0x7f2574105970 2024-05-04T12:08:21.333 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:20.522+0000 7f2579bcb700 1 -- 172.21.15.79:0/3829237662 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2574105970 msgr2=0x7f2574105d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:08:21.333 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:20.522+0000 7f2579bcb700 1 --2- 172.21.15.79:0/3829237662 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2574105970 0x7f2574105d90 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f25640098a0 tx=0x7f2564009bb0 comp rx=0 tx=0).stop 2024-05-04T12:08:21.333 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:20.522+0000 7f2579bcb700 1 -- 172.21.15.79:0/3829237662 shutdown_connections 2024-05-04T12:08:21.333 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:20.522+0000 7f2579bcb700 1 --2- 172.21.15.79:0/3829237662 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2574105970 0x7f2574105d90 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:08:21.333 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:20.522+0000 7f2579bcb700 1 -- 172.21.15.79:0/3829237662 >> 172.21.15.79:0/3829237662 conn(0x7f2574100190 msgr2=0x7f25741025d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:08:21.333 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:20.522+0000 7f2579bcb700 1 -- 172.21.15.79:0/3829237662 shutdown_connections 2024-05-04T12:08:21.333 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:20.522+0000 7f2579bcb700 1 -- 172.21.15.79:0/3829237662 wait complete. 2024-05-04T12:08:21.333 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:20.523+0000 7f2579bcb700 1 Processor -- start 2024-05-04T12:08:21.333 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:20.523+0000 7f2579bcb700 1 -- start start 2024-05-04T12:08:21.333 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:20.523+0000 7f2579bcb700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2574105970 0x7f2574190610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:08:21.334 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:20.523+0000 7f2579bcb700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2574190b50 con 0x7f2574105970 2024-05-04T12:08:21.334 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:20.523+0000 7f25737fe700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2574105970 0x7f2574190610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:08:21.334 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:20.523+0000 7f25737fe700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2574105970 0x7f2574190610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:49316/0 (socket says 172.21.15.79:49316) 2024-05-04T12:08:21.334 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:20.523+0000 7f25737fe700 1 -- 172.21.15.79:0/670064659 learned_addr learned my addr 172.21.15.79:0/670064659 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:08:21.334 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:20.524+0000 7f25737fe700 1 -- 172.21.15.79:0/670064659 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2564009550 con 0x7f2574105970 2024-05-04T12:08:21.334 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:20.524+0000 7f25737fe700 1 --2- 172.21.15.79:0/670064659 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2574105970 0x7f2574190610 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f2564000c00 tx=0x7f2564011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:08:21.334 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:20.524+0000 7f2570ff9700 1 -- 172.21.15.79:0/670064659 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2564003b90 con 0x7f2574105970 2024-05-04T12:08:21.334 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:20.524+0000 7f2579bcb700 1 -- 172.21.15.79:0/670064659 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f257418ec60 con 0x7f2574105970 2024-05-04T12:08:21.334 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:20.524+0000 7f2570ff9700 1 -- 172.21.15.79:0/670064659 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f25640248b0 con 0x7f2574105970 2024-05-04T12:08:21.334 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:20.524+0000 7f2570ff9700 1 -- 172.21.15.79:0/670064659 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f256401a8e0 con 0x7f2574105970 2024-05-04T12:08:21.334 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:20.524+0000 7f2579bcb700 1 -- 172.21.15.79:0/670064659 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f257418f100 con 0x7f2574105970 2024-05-04T12:08:21.334 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:20.525+0000 7f2570ff9700 1 -- 172.21.15.79:0/670064659 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 5) v1 ==== 45283+0+0 (secure 0 0 0) 0x7f256401ab50 con 0x7f2574105970 2024-05-04T12:08:21.334 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:20.525+0000 7f2579bcb700 1 -- 172.21.15.79:0/670064659 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25740623c0 con 0x7f2574105970 2024-05-04T12:08:21.334 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:20.525+0000 7f2570ff9700 1 --2- 172.21.15.79:0/670064659 >> [v2:172.21.15.79:6800/839275478,v1:172.21.15.79:6801/839275478] conn(0x7f255c038370 0x7f255c03a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:08:21.335 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:20.525+0000 7f2570ff9700 1 -- 172.21.15.79:0/670064659 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f256404bfe0 con 0x7f2574105970 2024-05-04T12:08:21.335 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:20.525+0000 7f2572ffd700 1 -- 172.21.15.79:0/670064659 >> [v2:172.21.15.79:6800/839275478,v1:172.21.15.79:6801/839275478] conn(0x7f255c038370 msgr2=0x7f255c03a830 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.79:6800/839275478 2024-05-04T12:08:21.335 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:20.525+0000 7f2572ffd700 1 --2- 172.21.15.79:0/670064659 >> [v2:172.21.15.79:6800/839275478,v1:172.21.15.79:6801/839275478] conn(0x7f255c038370 0x7f255c03a830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-04T12:08:21.335 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:20.530+0000 7f2570ff9700 1 -- 172.21.15.79:0/670064659 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f256401ae00 con 0x7f2574105970 2024-05-04T12:08:21.335 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:20.726+0000 7f2572ffd700 1 -- 172.21.15.79:0/670064659 >> [v2:172.21.15.79:6800/839275478,v1:172.21.15.79:6801/839275478] conn(0x7f255c038370 msgr2=0x7f255c03a830 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.79:6800/839275478 2024-05-04T12:08:21.335 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:20.726+0000 7f2572ffd700 1 --2- 172.21.15.79:0/670064659 >> [v2:172.21.15.79:6800/839275478,v1:172.21.15.79:6801/839275478] conn(0x7f255c038370 0x7f255c03a830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-04T12:08:21.335 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:20.811+0000 7f2579bcb700 1 -- 172.21.15.79:0/670064659 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f2574109910 con 0x7f2574105970 2024-05-04T12:08:21.335 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:20.811+0000 7f2570ff9700 1 -- 172.21.15.79:0/670064659 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+103 (secure 0 0 0) 0x7f25640172b0 con 0x7f2574105970 2024-05-04T12:08:21.335 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:20.813+0000 7f2579bcb700 1 -- 172.21.15.79:0/670064659 >> [v2:172.21.15.79:6800/839275478,v1:172.21.15.79:6801/839275478] conn(0x7f255c038370 msgr2=0x7f255c03a830 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-04T12:08:21.335 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:20.813+0000 7f2579bcb700 1 --2- 172.21.15.79:0/670064659 >> [v2:172.21.15.79:6800/839275478,v1:172.21.15.79:6801/839275478] conn(0x7f255c038370 0x7f255c03a830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:08:21.335 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:20.813+0000 7f2579bcb700 1 -- 172.21.15.79:0/670064659 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2574105970 msgr2=0x7f2574190610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:08:21.335 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:20.813+0000 7f2579bcb700 1 --2- 172.21.15.79:0/670064659 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2574105970 0x7f2574190610 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f2564000c00 tx=0x7f2564011ab0 comp rx=0 tx=0).stop 2024-05-04T12:08:21.335 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:20.813+0000 7f2579bcb700 1 -- 172.21.15.79:0/670064659 shutdown_connections 2024-05-04T12:08:21.335 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:20.813+0000 7f2579bcb700 1 --2- 172.21.15.79:0/670064659 >> [v2:172.21.15.79:6800/839275478,v1:172.21.15.79:6801/839275478] conn(0x7f255c038370 0x7f255c03a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:08:21.336 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:20.813+0000 7f2579bcb700 1 --2- 172.21.15.79:0/670064659 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2574105970 0x7f2574190610 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:08:21.336 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:20.813+0000 7f2579bcb700 1 -- 172.21.15.79:0/670064659 >> 172.21.15.79:0/670064659 conn(0x7f2574100190 msgr2=0x7f2574104dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:08:21.336 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:20.814+0000 7f2579bcb700 1 -- 172.21.15.79:0/670064659 shutdown_connections 2024-05-04T12:08:21.336 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:20.814+0000 7f2579bcb700 1 -- 172.21.15.79:0/670064659 wait complete. 2024-05-04T12:08:21.336 INFO:teuthology.orchestra.run.smithi079.stdout:Waiting for the mgr to restart... 2024-05-04T12:08:21.336 INFO:teuthology.orchestra.run.smithi079.stdout:Waiting for mgr epoch 5... 2024-05-04T12:08:21.617 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:08:21 smithi079 ceph-mon[27222]: from='client.? 172.21.15.79:0/670064659' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-05-04T12:08:27.370 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:08:26 smithi079 ceph-mon[27222]: Active manager daemon smithi079.dgcpwl restarted 2024-05-04T12:08:27.371 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:08:26 smithi079 ceph-mon[27222]: Activating manager daemon smithi079.dgcpwl 2024-05-04T12:08:27.371 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:08:26 smithi079 ceph-mon[27222]: osdmap e2: 0 total, 0 up, 0 in 2024-05-04T12:08:27.371 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:08:26 smithi079 ceph-mon[27222]: mgrmap e6: smithi079.dgcpwl(active, starting, since 0.00410293s) 2024-05-04T12:08:27.371 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:08:26 smithi079 ceph-mon[27222]: from='mgr.14120 172.21.15.79:0/2394226123' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "mon metadata", "id": "smithi079"}]: dispatch 2024-05-04T12:08:27.371 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:08:26 smithi079 ceph-mon[27222]: from='mgr.14120 172.21.15.79:0/2394226123' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "mgr metadata", "who": "smithi079.dgcpwl", "id": "smithi079.dgcpwl"}]: dispatch 2024-05-04T12:08:27.371 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:08:26 smithi079 ceph-mon[27222]: from='mgr.14120 172.21.15.79:0/2394226123' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-04T12:08:27.371 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:08:26 smithi079 ceph-mon[27222]: from='mgr.14120 172.21.15.79:0/2394226123' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-04T12:08:27.371 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:08:26 smithi079 ceph-mon[27222]: from='mgr.14120 172.21.15.79:0/2394226123' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-04T12:08:27.371 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:08:26 smithi079 ceph-mon[27222]: Manager daemon smithi079.dgcpwl is now available 2024-05-04T12:08:27.371 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:08:26 smithi079 ceph-mon[27222]: Found migration_current of "None". Setting to last migration. 2024-05-04T12:08:27.371 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:08:26 smithi079 ceph-mon[27222]: from='mgr.14120 172.21.15.79:0/2394226123' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:08:27.371 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:08:26 smithi079 ceph-mon[27222]: from='mgr.14120 172.21.15.79:0/2394226123' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:08:27.371 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:08:26 smithi079 ceph-mon[27222]: from='mgr.14120 172.21.15.79:0/2394226123' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-04T12:08:27.372 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:08:26 smithi079 ceph-mon[27222]: from='mgr.14120 172.21.15.79:0/2394226123' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-04T12:08:27.372 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:08:26 smithi079 ceph-mon[27222]: from='mgr.14120 172.21.15.79:0/2394226123' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-04T12:08:27.372 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:08:26 smithi079 ceph-mon[27222]: from='mgr.14120 172.21.15.79:0/2394226123' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi079.dgcpwl/mirror_snapshot_schedule"}]: dispatch 2024-05-04T12:08:27.372 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:08:26 smithi079 ceph-mon[27222]: from='mgr.14120 172.21.15.79:0/2394226123' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi079.dgcpwl/trash_purge_schedule"}]: dispatch 2024-05-04T12:08:27.372 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:08:26 smithi079 ceph-mon[27222]: from='mgr.14120 172.21.15.79:0/2394226123' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:08:27.372 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:08:26 smithi079 ceph-mon[27222]: from='mgr.14120 172.21.15.79:0/2394226123' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:08:27.583 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout { 2024-05-04T12:08:27.583 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-05-04T12:08:27.583 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "initialized": true 2024-05-04T12:08:27.583 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout } 2024-05-04T12:08:27.583 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:21.717+0000 7fbb3f4aa700 1 Processor -- start 2024-05-04T12:08:27.583 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:21.717+0000 7fbb3f4aa700 1 -- start start 2024-05-04T12:08:27.584 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:21.717+0000 7fbb3f4aa700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbb38107020 0x7fbb38109410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:08:27.584 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:21.717+0000 7fbb3f4aa700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb38079e30 con 0x7fbb38107020 2024-05-04T12:08:27.584 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:21.717+0000 7fbb3d246700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbb38107020 0x7fbb38109410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:08:27.584 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:21.717+0000 7fbb3d246700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbb38107020 0x7fbb38109410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:49318/0 (socket says 172.21.15.79:49318) 2024-05-04T12:08:27.584 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:21.717+0000 7fbb3d246700 1 -- 172.21.15.79:0/3481077430 learned_addr learned my addr 172.21.15.79:0/3481077430 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:08:27.584 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:21.718+0000 7fbb3d246700 1 -- 172.21.15.79:0/3481077430 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb38109950 con 0x7fbb38107020 2024-05-04T12:08:27.584 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:21.718+0000 7fbb3d246700 1 --2- 172.21.15.79:0/3481077430 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbb38107020 0x7fbb38109410 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fbb280098a0 tx=0x7fbb28009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc9cc6d6ff14f1ee server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:08:27.584 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:21.718+0000 7fbb2ffff700 1 -- 172.21.15.79:0/3481077430 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbb28004180 con 0x7fbb38107020 2024-05-04T12:08:27.584 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:21.718+0000 7fbb2ffff700 1 -- 172.21.15.79:0/3481077430 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fbb280042e0 con 0x7fbb38107020 2024-05-04T12:08:27.584 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:21.718+0000 7fbb2ffff700 1 -- 172.21.15.79:0/3481077430 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbb28011a50 con 0x7fbb38107020 2024-05-04T12:08:27.584 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:21.719+0000 7fbb3f4aa700 1 -- 172.21.15.79:0/3481077430 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbb38107020 msgr2=0x7fbb38109410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:08:27.584 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:21.719+0000 7fbb3f4aa700 1 --2- 172.21.15.79:0/3481077430 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbb38107020 0x7fbb38109410 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fbb280098a0 tx=0x7fbb28009bb0 comp rx=0 tx=0).stop 2024-05-04T12:08:27.584 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:21.719+0000 7fbb3f4aa700 1 -- 172.21.15.79:0/3481077430 shutdown_connections 2024-05-04T12:08:27.585 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:21.719+0000 7fbb3f4aa700 1 --2- 172.21.15.79:0/3481077430 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbb38107020 0x7fbb38109410 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:08:27.585 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:21.719+0000 7fbb3f4aa700 1 -- 172.21.15.79:0/3481077430 >> 172.21.15.79:0/3481077430 conn(0x7fbb381009b0 msgr2=0x7fbb38102e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:08:27.585 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:21.719+0000 7fbb3f4aa700 1 -- 172.21.15.79:0/3481077430 shutdown_connections 2024-05-04T12:08:27.585 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:21.719+0000 7fbb3f4aa700 1 -- 172.21.15.79:0/3481077430 wait complete. 2024-05-04T12:08:27.585 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:21.719+0000 7fbb3f4aa700 1 Processor -- start 2024-05-04T12:08:27.585 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:21.719+0000 7fbb3f4aa700 1 -- start start 2024-05-04T12:08:27.585 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:21.720+0000 7fbb3f4aa700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbb38107020 0x7fbb38106980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:08:27.585 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:21.720+0000 7fbb3f4aa700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb38106ec0 con 0x7fbb38107020 2024-05-04T12:08:27.585 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:21.720+0000 7fbb3d246700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbb38107020 0x7fbb38106980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:08:27.585 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:21.720+0000 7fbb3d246700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbb38107020 0x7fbb38106980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:49330/0 (socket says 172.21.15.79:49330) 2024-05-04T12:08:27.585 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:21.720+0000 7fbb3d246700 1 -- 172.21.15.79:0/4014899406 learned_addr learned my addr 172.21.15.79:0/4014899406 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:08:27.585 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:21.720+0000 7fbb3d246700 1 -- 172.21.15.79:0/4014899406 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb28009550 con 0x7fbb38107020 2024-05-04T12:08:27.585 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:21.720+0000 7fbb3d246700 1 --2- 172.21.15.79:0/4014899406 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbb38107020 0x7fbb38106980 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fbb28003f80 tx=0x7fbb28011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:08:27.585 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:21.720+0000 7fbb2e7fc700 1 -- 172.21.15.79:0/4014899406 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbb28003a80 con 0x7fbb38107020 2024-05-04T12:08:27.586 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:21.720+0000 7fbb2e7fc700 1 -- 172.21.15.79:0/4014899406 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fbb28003be0 con 0x7fbb38107020 2024-05-04T12:08:27.586 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:21.720+0000 7fbb3f4aa700 1 -- 172.21.15.79:0/4014899406 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb38104fd0 con 0x7fbb38107020 2024-05-04T12:08:27.586 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:21.720+0000 7fbb2e7fc700 1 -- 172.21.15.79:0/4014899406 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbb28019820 con 0x7fbb38107020 2024-05-04T12:08:27.586 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:21.720+0000 7fbb3f4aa700 1 -- 172.21.15.79:0/4014899406 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb38105470 con 0x7fbb38107020 2024-05-04T12:08:27.586 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:21.721+0000 7fbb2e7fc700 1 -- 172.21.15.79:0/4014899406 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 5) v1 ==== 45283+0+0 (secure 0 0 0) 0x7fbb28019aa0 con 0x7fbb38107020 2024-05-04T12:08:27.586 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:21.721+0000 7fbb2e7fc700 1 --2- 172.21.15.79:0/4014899406 >> [v2:172.21.15.79:6800/839275478,v1:172.21.15.79:6801/839275478] conn(0x7fbb240382d0 0x7fbb2403a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:08:27.586 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:21.721+0000 7fbb2e7fc700 1 -- 172.21.15.79:0/4014899406 --> [v2:172.21.15.79:6800/839275478,v1:172.21.15.79:6801/839275478] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbb2403aea0 con 0x7fbb240382d0 2024-05-04T12:08:27.586 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:21.721+0000 7fbb2e7fc700 1 -- 172.21.15.79:0/4014899406 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fbb2804c590 con 0x7fbb38107020 2024-05-04T12:08:27.586 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:21.721+0000 7fbb3ca45700 1 -- 172.21.15.79:0/4014899406 >> [v2:172.21.15.79:6800/839275478,v1:172.21.15.79:6801/839275478] conn(0x7fbb240382d0 msgr2=0x7fbb2403a790 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.79:6800/839275478 2024-05-04T12:08:27.586 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:21.721+0000 7fbb3ca45700 1 --2- 172.21.15.79:0/4014899406 >> [v2:172.21.15.79:6800/839275478,v1:172.21.15.79:6801/839275478] conn(0x7fbb240382d0 0x7fbb2403a790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-04T12:08:27.586 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:21.922+0000 7fbb3ca45700 1 -- 172.21.15.79:0/4014899406 >> [v2:172.21.15.79:6800/839275478,v1:172.21.15.79:6801/839275478] conn(0x7fbb240382d0 msgr2=0x7fbb2403a790 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.79:6800/839275478 2024-05-04T12:08:27.586 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:21.922+0000 7fbb3ca45700 1 --2- 172.21.15.79:0/4014899406 >> [v2:172.21.15.79:6800/839275478,v1:172.21.15.79:6801/839275478] conn(0x7fbb240382d0 0x7fbb2403a790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-04T12:08:27.586 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:22.323+0000 7fbb3ca45700 1 -- 172.21.15.79:0/4014899406 >> [v2:172.21.15.79:6800/839275478,v1:172.21.15.79:6801/839275478] conn(0x7fbb240382d0 msgr2=0x7fbb2403a790 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.79:6800/839275478 2024-05-04T12:08:27.586 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:22.323+0000 7fbb3ca45700 1 --2- 172.21.15.79:0/4014899406 >> [v2:172.21.15.79:6800/839275478,v1:172.21.15.79:6801/839275478] conn(0x7fbb240382d0 0x7fbb2403a790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-04T12:08:27.587 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:23.124+0000 7fbb3ca45700 1 -- 172.21.15.79:0/4014899406 >> [v2:172.21.15.79:6800/839275478,v1:172.21.15.79:6801/839275478] conn(0x7fbb240382d0 msgr2=0x7fbb2403a790 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.79:6800/839275478 2024-05-04T12:08:27.587 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:23.124+0000 7fbb3ca45700 1 --2- 172.21.15.79:0/4014899406 >> [v2:172.21.15.79:6800/839275478,v1:172.21.15.79:6801/839275478] conn(0x7fbb240382d0 0x7fbb2403a790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-04T12:08:27.587 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:24.725+0000 7fbb3ca45700 1 -- 172.21.15.79:0/4014899406 >> [v2:172.21.15.79:6800/839275478,v1:172.21.15.79:6801/839275478] conn(0x7fbb240382d0 msgr2=0x7fbb2403a790 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.79:6800/839275478 2024-05-04T12:08:27.587 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:24.725+0000 7fbb3ca45700 1 --2- 172.21.15.79:0/4014899406 >> [v2:172.21.15.79:6800/839275478,v1:172.21.15.79:6801/839275478] conn(0x7fbb240382d0 0x7fbb2403a790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-05-04T12:08:27.587 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:26.033+0000 7fbb2e7fc700 1 -- 172.21.15.79:0/4014899406 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mgrmap(e 6) v1 ==== 45050+0+0 (secure 0 0 0) 0x7fbb28016d40 con 0x7fbb38107020 2024-05-04T12:08:27.587 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:26.033+0000 7fbb2e7fc700 1 -- 172.21.15.79:0/4014899406 >> [v2:172.21.15.79:6800/839275478,v1:172.21.15.79:6801/839275478] conn(0x7fbb240382d0 msgr2=0x7fbb2403a790 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-04T12:08:27.587 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:26.033+0000 7fbb2e7fc700 1 --2- 172.21.15.79:0/4014899406 >> [v2:172.21.15.79:6800/839275478,v1:172.21.15.79:6801/839275478] conn(0x7fbb240382d0 0x7fbb2403a790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:08:27.587 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:27.034+0000 7fbb2e7fc700 1 -- 172.21.15.79:0/4014899406 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mgrmap(e 7) v1 ==== 45177+0+0 (secure 0 0 0) 0x7fbb28019d50 con 0x7fbb38107020 2024-05-04T12:08:27.587 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:27.034+0000 7fbb2e7fc700 1 --2- 172.21.15.79:0/4014899406 >> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] conn(0x7fbb2403bc00 0x7fbb2403dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:08:27.587 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:27.034+0000 7fbb2e7fc700 1 -- 172.21.15.79:0/4014899406 --> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbb2403aea0 con 0x7fbb2403bc00 2024-05-04T12:08:27.587 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:27.034+0000 7fbb3ca45700 1 --2- 172.21.15.79:0/4014899406 >> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] conn(0x7fbb2403bc00 0x7fbb2403dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:08:27.587 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:27.035+0000 7fbb3ca45700 1 --2- 172.21.15.79:0/4014899406 >> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] conn(0x7fbb2403bc00 0x7fbb2403dff0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fbb34003a10 tx=0x7fbb34009290 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:08:27.587 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:27.036+0000 7fbb2e7fc700 1 -- 172.21.15.79:0/4014899406 <== mgr.14120 v2:172.21.15.79:6800/1620705054 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6910 (secure 0 0 0) 0x7fbb2403aea0 con 0x7fbb2403bc00 2024-05-04T12:08:27.587 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:27.042+0000 7fbb3f4aa700 1 -- 172.21.15.79:0/4014899406 --> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7fbb38106520 con 0x7fbb2403bc00 2024-05-04T12:08:27.588 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:27.042+0000 7fbb2e7fc700 1 -- 172.21.15.79:0/4014899406 <== mgr.14120 v2:172.21.15.79:6800/1620705054 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7fbb38106520 con 0x7fbb2403bc00 2024-05-04T12:08:27.588 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:27.042+0000 7fbb3f4aa700 1 -- 172.21.15.79:0/4014899406 >> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] conn(0x7fbb2403bc00 msgr2=0x7fbb2403dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:08:27.588 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:27.042+0000 7fbb3f4aa700 1 --2- 172.21.15.79:0/4014899406 >> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] conn(0x7fbb2403bc00 0x7fbb2403dff0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fbb34003a10 tx=0x7fbb34009290 comp rx=0 tx=0).stop 2024-05-04T12:08:27.588 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:27.042+0000 7fbb3f4aa700 1 -- 172.21.15.79:0/4014899406 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbb38107020 msgr2=0x7fbb38106980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:08:27.588 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:27.042+0000 7fbb3f4aa700 1 --2- 172.21.15.79:0/4014899406 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbb38107020 0x7fbb38106980 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fbb28003f80 tx=0x7fbb28011ab0 comp rx=0 tx=0).stop 2024-05-04T12:08:27.588 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:27.042+0000 7fbb3f4aa700 1 -- 172.21.15.79:0/4014899406 shutdown_connections 2024-05-04T12:08:27.588 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:27.042+0000 7fbb3f4aa700 1 --2- 172.21.15.79:0/4014899406 >> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] conn(0x7fbb2403bc00 0x7fbb2403dff0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:08:27.588 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:27.042+0000 7fbb3f4aa700 1 --2- 172.21.15.79:0/4014899406 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbb38107020 0x7fbb38106980 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:08:27.588 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:27.042+0000 7fbb3f4aa700 1 --2- 172.21.15.79:0/4014899406 >> [v2:172.21.15.79:6800/839275478,v1:172.21.15.79:6801/839275478] conn(0x7fbb240382d0 0x7fbb2403a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:08:27.588 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:27.042+0000 7fbb3f4aa700 1 -- 172.21.15.79:0/4014899406 >> 172.21.15.79:0/4014899406 conn(0x7fbb381009b0 msgr2=0x7fbb38102060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:08:27.588 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:27.042+0000 7fbb3f4aa700 1 -- 172.21.15.79:0/4014899406 shutdown_connections 2024-05-04T12:08:27.588 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:27.042+0000 7fbb3f4aa700 1 -- 172.21.15.79:0/4014899406 wait complete. 2024-05-04T12:08:27.588 INFO:teuthology.orchestra.run.smithi079.stdout:mgr epoch 5 is available 2024-05-04T12:08:27.588 INFO:teuthology.orchestra.run.smithi079.stdout:Setting orchestrator backend to cephadm... 2024-05-04T12:08:28.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:08:28 smithi079 ceph-mon[27222]: [04/May/2024:12:08:26] ENGINE Bus STARTING 2024-05-04T12:08:28.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:08:28 smithi079 ceph-mon[27222]: [04/May/2024:12:08:26] ENGINE Serving on http://172.21.15.79:8765 2024-05-04T12:08:28.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:08:28 smithi079 ceph-mon[27222]: [04/May/2024:12:08:26] ENGINE Serving on https://172.21.15.79:7150 2024-05-04T12:08:28.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:08:28 smithi079 ceph-mon[27222]: [04/May/2024:12:08:26] ENGINE Bus STARTED 2024-05-04T12:08:28.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:08:28 smithi079 ceph-mon[27222]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-05-04T12:08:28.446 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:08:28 smithi079 ceph-mon[27222]: mgrmap e7: smithi079.dgcpwl(active, since 1.00555s) 2024-05-04T12:08:28.446 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:08:28 smithi079 ceph-mon[27222]: from='mgr.14120 172.21.15.79:0/2394226123' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-04T12:08:28.446 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:08:28 smithi079 ceph-mon[27222]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-05-04T12:08:28.768 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:27.957+0000 7f062295a700 1 Processor -- start 2024-05-04T12:08:28.768 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:27.957+0000 7f062295a700 1 -- start start 2024-05-04T12:08:28.769 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:27.957+0000 7f062295a700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f061c106560 0x7f061c106980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:08:28.769 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:27.957+0000 7f062295a700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f061c106f50 con 0x7f061c106560 2024-05-04T12:08:28.769 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:27.957+0000 7f061bfff700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f061c106560 0x7f061c106980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:08:28.769 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:27.957+0000 7f061bfff700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f061c106560 0x7f061c106980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:52416/0 (socket says 172.21.15.79:52416) 2024-05-04T12:08:28.769 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:27.957+0000 7f061bfff700 1 -- 172.21.15.79:0/1718286581 learned_addr learned my addr 172.21.15.79:0/1718286581 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:08:28.769 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:27.958+0000 7f061bfff700 1 -- 172.21.15.79:0/1718286581 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f061c107760 con 0x7f061c106560 2024-05-04T12:08:28.769 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:27.958+0000 7f061bfff700 1 --2- 172.21.15.79:0/1718286581 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f061c106560 0x7f061c106980 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f060c0098a0 tx=0x7f060c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cf8d9a4b750c541 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:08:28.769 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:27.958+0000 7f061affd700 1 -- 172.21.15.79:0/1718286581 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f060c004180 con 0x7f061c106560 2024-05-04T12:08:28.769 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:27.958+0000 7f061affd700 1 -- 172.21.15.79:0/1718286581 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f060c0042e0 con 0x7f061c106560 2024-05-04T12:08:28.769 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:27.958+0000 7f061affd700 1 -- 172.21.15.79:0/1718286581 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f060c011a50 con 0x7f061c106560 2024-05-04T12:08:28.769 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:27.958+0000 7f062295a700 1 -- 172.21.15.79:0/1718286581 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f061c106560 msgr2=0x7f061c106980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:08:28.769 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:27.958+0000 7f062295a700 1 --2- 172.21.15.79:0/1718286581 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f061c106560 0x7f061c106980 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f060c0098a0 tx=0x7f060c009bb0 comp rx=0 tx=0).stop 2024-05-04T12:08:28.769 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:27.958+0000 7f062295a700 1 -- 172.21.15.79:0/1718286581 shutdown_connections 2024-05-04T12:08:28.770 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:27.958+0000 7f062295a700 1 --2- 172.21.15.79:0/1718286581 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f061c106560 0x7f061c106980 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:08:28.770 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:27.958+0000 7f062295a700 1 -- 172.21.15.79:0/1718286581 >> 172.21.15.79:0/1718286581 conn(0x7f061c101b40 msgr2=0x7f061c103f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:08:28.770 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:27.958+0000 7f062295a700 1 -- 172.21.15.79:0/1718286581 shutdown_connections 2024-05-04T12:08:28.770 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:27.958+0000 7f062295a700 1 -- 172.21.15.79:0/1718286581 wait complete. 2024-05-04T12:08:28.770 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:27.959+0000 7f062295a700 1 Processor -- start 2024-05-04T12:08:28.770 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:27.959+0000 7f062295a700 1 -- start start 2024-05-04T12:08:28.770 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:27.959+0000 7f062295a700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f061c106560 0x7f061c07bc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:08:28.770 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:27.959+0000 7f062295a700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f061c07c170 con 0x7f061c106560 2024-05-04T12:08:28.770 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:27.959+0000 7f061bfff700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f061c106560 0x7f061c07bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:08:28.770 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:27.959+0000 7f061bfff700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f061c106560 0x7f061c07bc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:52424/0 (socket says 172.21.15.79:52424) 2024-05-04T12:08:28.770 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:27.959+0000 7f061bfff700 1 -- 172.21.15.79:0/2943490992 learned_addr learned my addr 172.21.15.79:0/2943490992 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:08:28.770 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:27.960+0000 7f061bfff700 1 -- 172.21.15.79:0/2943490992 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f060c009550 con 0x7f061c106560 2024-05-04T12:08:28.770 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:27.960+0000 7f061bfff700 1 --2- 172.21.15.79:0/2943490992 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f061c106560 0x7f061c07bc30 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f060c003de0 tx=0x7f060c011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:08:28.770 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:27.960+0000 7f06197fa700 1 -- 172.21.15.79:0/2943490992 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f060c003930 con 0x7f061c106560 2024-05-04T12:08:28.771 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:27.960+0000 7f06197fa700 1 -- 172.21.15.79:0/2943490992 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f060c003a90 con 0x7f061c106560 2024-05-04T12:08:28.771 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:27.960+0000 7f062295a700 1 -- 172.21.15.79:0/2943490992 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f061c07a280 con 0x7f061c106560 2024-05-04T12:08:28.771 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:27.960+0000 7f06197fa700 1 -- 172.21.15.79:0/2943490992 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f060c0196c0 con 0x7f061c106560 2024-05-04T12:08:28.771 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:27.960+0000 7f062295a700 1 -- 172.21.15.79:0/2943490992 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f061c07a720 con 0x7f061c106560 2024-05-04T12:08:28.771 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:27.961+0000 7f06197fa700 1 -- 172.21.15.79:0/2943490992 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 7) v1 ==== 45177+0+0 (secure 0 0 0) 0x7f060c019930 con 0x7f061c106560 2024-05-04T12:08:28.771 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:27.961+0000 7f06197fa700 1 --2- 172.21.15.79:0/2943490992 >> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] conn(0x7f0604038250 0x7f060403a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:08:28.771 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:27.961+0000 7f06197fa700 1 -- 172.21.15.79:0/2943490992 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f060c04d140 con 0x7f061c106560 2024-05-04T12:08:28.771 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:27.961+0000 7f061b7fe700 1 --2- 172.21.15.79:0/2943490992 >> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] conn(0x7f0604038250 0x7f060403a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:08:28.771 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:27.961+0000 7f062295a700 1 -- 172.21.15.79:0/2943490992 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f061c0623c0 con 0x7f061c106560 2024-05-04T12:08:28.771 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:27.961+0000 7f061b7fe700 1 --2- 172.21.15.79:0/2943490992 >> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] conn(0x7f0604038250 0x7f060403a710 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f0610006fd0 tx=0x7f0610006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:08:28.771 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:27.966+0000 7f06197fa700 1 -- 172.21.15.79:0/2943490992 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f060c019c30 con 0x7f061c106560 2024-05-04T12:08:28.771 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:28.040+0000 7f06197fa700 1 -- 172.21.15.79:0/2943490992 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7f060c00fd80 con 0x7f061c106560 2024-05-04T12:08:28.771 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:28.168+0000 7f062295a700 1 -- 172.21.15.79:0/2943490992 --> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7f061c10a960 con 0x7f0604038250 2024-05-04T12:08:28.772 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:28.177+0000 7f06197fa700 1 -- 172.21.15.79:0/2943490992 <== mgr.14120 v2:172.21.15.79:6800/1620705054 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f061c10a960 con 0x7f0604038250 2024-05-04T12:08:28.772 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:28.180+0000 7f062295a700 1 -- 172.21.15.79:0/2943490992 >> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] conn(0x7f0604038250 msgr2=0x7f060403a710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:08:28.772 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:28.180+0000 7f062295a700 1 --2- 172.21.15.79:0/2943490992 >> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] conn(0x7f0604038250 0x7f060403a710 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f0610006fd0 tx=0x7f0610006e40 comp rx=0 tx=0).stop 2024-05-04T12:08:28.772 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:28.180+0000 7f062295a700 1 -- 172.21.15.79:0/2943490992 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f061c106560 msgr2=0x7f061c07bc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:08:28.772 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:28.180+0000 7f062295a700 1 --2- 172.21.15.79:0/2943490992 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f061c106560 0x7f061c07bc30 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f060c003de0 tx=0x7f060c011ab0 comp rx=0 tx=0).stop 2024-05-04T12:08:28.772 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:28.180+0000 7f062295a700 1 -- 172.21.15.79:0/2943490992 shutdown_connections 2024-05-04T12:08:28.772 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:28.180+0000 7f062295a700 1 --2- 172.21.15.79:0/2943490992 >> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] conn(0x7f0604038250 0x7f060403a710 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:08:28.772 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:28.180+0000 7f062295a700 1 --2- 172.21.15.79:0/2943490992 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f061c106560 0x7f061c07bc30 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:08:28.772 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:28.180+0000 7f062295a700 1 -- 172.21.15.79:0/2943490992 >> 172.21.15.79:0/2943490992 conn(0x7f061c101b40 msgr2=0x7f061c102900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:08:28.772 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:28.180+0000 7f062295a700 1 -- 172.21.15.79:0/2943490992 shutdown_connections 2024-05-04T12:08:28.772 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:28.180+0000 7f062295a700 1 -- 172.21.15.79:0/2943490992 wait complete. 2024-05-04T12:08:29.394 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:08:29 smithi079 ceph-mon[27222]: mgrmap e8: smithi079.dgcpwl(active, since 2s) 2024-05-04T12:08:29.394 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:08:29 smithi079 ceph-mon[27222]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-05-04T12:08:29.395 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:08:29 smithi079 ceph-mon[27222]: from='mgr.14120 172.21.15.79:0/2394226123' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:08:29.395 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:08:29 smithi079 ceph-mon[27222]: from='mgr.14120 172.21.15.79:0/2394226123' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-04T12:08:29.966 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout value unchanged 2024-05-04T12:08:29.967 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:29.197+0000 7fb622fb1700 1 Processor -- start 2024-05-04T12:08:29.967 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:29.197+0000 7fb622fb1700 1 -- start start 2024-05-04T12:08:29.967 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:29.197+0000 7fb622fb1700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb61c07bea0 0x7fb61c07a300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:08:29.967 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:29.197+0000 7fb622fb1700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb61c07a8d0 con 0x7fb61c07bea0 2024-05-04T12:08:29.967 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:29.197+0000 7fb620d4d700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb61c07bea0 0x7fb61c07a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:08:29.967 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:29.197+0000 7fb620d4d700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb61c07bea0 0x7fb61c07a300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:52440/0 (socket says 172.21.15.79:52440) 2024-05-04T12:08:29.967 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:29.197+0000 7fb620d4d700 1 -- 172.21.15.79:0/69986182 learned_addr learned my addr 172.21.15.79:0/69986182 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:08:29.967 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:29.197+0000 7fb620d4d700 1 -- 172.21.15.79:0/69986182 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb61c07aa10 con 0x7fb61c07bea0 2024-05-04T12:08:29.967 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:29.198+0000 7fb620d4d700 1 --2- 172.21.15.79:0/69986182 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb61c07bea0 0x7fb61c07a300 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fb6100098a0 tx=0x7fb610009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fdfc35d97a8aa0e server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:08:29.967 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:29.198+0000 7fb61b7fe700 1 -- 172.21.15.79:0/69986182 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb610004180 con 0x7fb61c07bea0 2024-05-04T12:08:29.967 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:29.198+0000 7fb61b7fe700 1 -- 172.21.15.79:0/69986182 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fb6100042e0 con 0x7fb61c07bea0 2024-05-04T12:08:29.967 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:29.198+0000 7fb61b7fe700 1 -- 172.21.15.79:0/69986182 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb610011a50 con 0x7fb61c07bea0 2024-05-04T12:08:29.968 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:29.198+0000 7fb622fb1700 1 -- 172.21.15.79:0/69986182 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb61c07bea0 msgr2=0x7fb61c07a300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:08:29.968 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:29.198+0000 7fb622fb1700 1 --2- 172.21.15.79:0/69986182 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb61c07bea0 0x7fb61c07a300 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fb6100098a0 tx=0x7fb610009bb0 comp rx=0 tx=0).stop 2024-05-04T12:08:29.968 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:29.198+0000 7fb622fb1700 1 -- 172.21.15.79:0/69986182 shutdown_connections 2024-05-04T12:08:29.968 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:29.198+0000 7fb622fb1700 1 --2- 172.21.15.79:0/69986182 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb61c07bea0 0x7fb61c07a300 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:08:29.968 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:29.198+0000 7fb622fb1700 1 -- 172.21.15.79:0/69986182 >> 172.21.15.79:0/69986182 conn(0x7fb61c101ad0 msgr2=0x7fb61c103f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:08:29.968 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:29.198+0000 7fb622fb1700 1 -- 172.21.15.79:0/69986182 shutdown_connections 2024-05-04T12:08:29.968 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:29.198+0000 7fb622fb1700 1 -- 172.21.15.79:0/69986182 wait complete. 2024-05-04T12:08:29.968 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:29.199+0000 7fb622fb1700 1 Processor -- start 2024-05-04T12:08:29.968 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:29.199+0000 7fb622fb1700 1 -- start start 2024-05-04T12:08:29.968 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:29.199+0000 7fb622fb1700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb61c07bea0 0x7fb61c19c420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:08:29.968 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:29.199+0000 7fb622fb1700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb61c19c960 con 0x7fb61c07bea0 2024-05-04T12:08:29.968 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:29.199+0000 7fb620d4d700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb61c07bea0 0x7fb61c19c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:08:29.968 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:29.199+0000 7fb620d4d700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb61c07bea0 0x7fb61c19c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:52450/0 (socket says 172.21.15.79:52450) 2024-05-04T12:08:29.968 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:29.199+0000 7fb620d4d700 1 -- 172.21.15.79:0/1667394464 learned_addr learned my addr 172.21.15.79:0/1667394464 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:08:29.968 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:29.199+0000 7fb620d4d700 1 -- 172.21.15.79:0/1667394464 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb610009550 con 0x7fb61c07bea0 2024-05-04T12:08:29.969 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:29.200+0000 7fb620d4d700 1 --2- 172.21.15.79:0/1667394464 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb61c07bea0 0x7fb61c19c420 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fb610009520 tx=0x7fb610011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:08:29.969 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:29.200+0000 7fb619ffb700 1 -- 172.21.15.79:0/1667394464 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb610003a80 con 0x7fb61c07bea0 2024-05-04T12:08:29.969 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:29.200+0000 7fb622fb1700 1 -- 172.21.15.79:0/1667394464 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb61c19cb60 con 0x7fb61c07bea0 2024-05-04T12:08:29.969 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:29.200+0000 7fb619ffb700 1 -- 172.21.15.79:0/1667394464 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fb610003be0 con 0x7fb61c07bea0 2024-05-04T12:08:29.969 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:29.200+0000 7fb619ffb700 1 -- 172.21.15.79:0/1667394464 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb6100197b0 con 0x7fb61c07bea0 2024-05-04T12:08:29.969 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:29.200+0000 7fb622fb1700 1 -- 172.21.15.79:0/1667394464 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb61c19d000 con 0x7fb61c07bea0 2024-05-04T12:08:29.969 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:29.201+0000 7fb619ffb700 1 -- 172.21.15.79:0/1667394464 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7fb610019a70 con 0x7fb61c07bea0 2024-05-04T12:08:29.969 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:29.201+0000 7fb619ffb700 1 --2- 172.21.15.79:0/1667394464 >> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] conn(0x7fb60c0382d0 0x7fb60c03a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:08:29.969 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:29.201+0000 7fb622fb1700 1 -- 172.21.15.79:0/1667394464 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb61c10c870 con 0x7fb61c07bea0 2024-05-04T12:08:29.969 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:29.201+0000 7fb619ffb700 1 -- 172.21.15.79:0/1667394464 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fb61004c700 con 0x7fb61c07bea0 2024-05-04T12:08:29.969 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:29.201+0000 7fb61bfff700 1 --2- 172.21.15.79:0/1667394464 >> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] conn(0x7fb60c0382d0 0x7fb60c03a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:08:29.969 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:29.201+0000 7fb61bfff700 1 --2- 172.21.15.79:0/1667394464 >> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] conn(0x7fb60c0382d0 0x7fb60c03a790 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fb608006fd0 tx=0x7fb608006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:08:29.969 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:29.206+0000 7fb619ffb700 1 -- 172.21.15.79:0/1667394464 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fb61002c3a0 con 0x7fb61c07bea0 2024-05-04T12:08:29.969 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:29.395+0000 7fb622fb1700 1 -- 172.21.15.79:0/1667394464 --> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7fb61c103e40 con 0x7fb60c0382d0 2024-05-04T12:08:29.970 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:29.396+0000 7fb619ffb700 1 -- 172.21.15.79:0/1667394464 <== mgr.14120 v2:172.21.15.79:6800/1620705054 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7fb61c103e40 con 0x7fb60c0382d0 2024-05-04T12:08:29.970 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:29.398+0000 7fb622fb1700 1 -- 172.21.15.79:0/1667394464 >> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] conn(0x7fb60c0382d0 msgr2=0x7fb60c03a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:08:29.970 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:29.398+0000 7fb622fb1700 1 --2- 172.21.15.79:0/1667394464 >> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] conn(0x7fb60c0382d0 0x7fb60c03a790 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fb608006fd0 tx=0x7fb608006e40 comp rx=0 tx=0).stop 2024-05-04T12:08:29.970 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:29.398+0000 7fb622fb1700 1 -- 172.21.15.79:0/1667394464 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb61c07bea0 msgr2=0x7fb61c19c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:08:29.970 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:29.398+0000 7fb622fb1700 1 --2- 172.21.15.79:0/1667394464 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb61c07bea0 0x7fb61c19c420 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fb610009520 tx=0x7fb610011ab0 comp rx=0 tx=0).stop 2024-05-04T12:08:29.970 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:29.398+0000 7fb622fb1700 1 -- 172.21.15.79:0/1667394464 shutdown_connections 2024-05-04T12:08:29.970 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:29.398+0000 7fb622fb1700 1 --2- 172.21.15.79:0/1667394464 >> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] conn(0x7fb60c0382d0 0x7fb60c03a790 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:08:29.970 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:29.398+0000 7fb622fb1700 1 --2- 172.21.15.79:0/1667394464 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb61c07bea0 0x7fb61c19c420 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:08:29.970 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:29.398+0000 7fb622fb1700 1 -- 172.21.15.79:0/1667394464 >> 172.21.15.79:0/1667394464 conn(0x7fb61c101ad0 msgr2=0x7fb61c103220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:08:29.970 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:29.398+0000 7fb622fb1700 1 -- 172.21.15.79:0/1667394464 shutdown_connections 2024-05-04T12:08:29.970 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:29.398+0000 7fb622fb1700 1 -- 172.21.15.79:0/1667394464 wait complete. 2024-05-04T12:08:29.970 INFO:teuthology.orchestra.run.smithi079.stdout:Generating ssh key... 2024-05-04T12:08:30.217 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:08:30 smithi079 ceph-mon[27222]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-05-04T12:08:31.314 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:30.392+0000 7f0952810700 1 Processor -- start 2024-05-04T12:08:31.314 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:30.392+0000 7f0952810700 1 -- start start 2024-05-04T12:08:31.314 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:30.392+0000 7f0952810700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f094c106ad0 0x7f094c106ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:08:31.315 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:30.392+0000 7f0952810700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f094c1074c0 con 0x7f094c106ad0 2024-05-04T12:08:31.315 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:30.392+0000 7f094bfff700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f094c106ad0 0x7f094c106ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:08:31.315 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:30.392+0000 7f094bfff700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f094c106ad0 0x7f094c106ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:52454/0 (socket says 172.21.15.79:52454) 2024-05-04T12:08:31.315 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:30.392+0000 7f094bfff700 1 -- 172.21.15.79:0/492095081 learned_addr learned my addr 172.21.15.79:0/492095081 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:08:31.315 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:30.392+0000 7f094bfff700 1 -- 172.21.15.79:0/492095081 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f094c1056a0 con 0x7f094c106ad0 2024-05-04T12:08:31.315 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:30.393+0000 7f094bfff700 1 --2- 172.21.15.79:0/492095081 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f094c106ad0 0x7f094c106ef0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f093c0098a0 tx=0x7f093c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6887ff4bcef6287 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:08:31.315 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:30.393+0000 7f094affd700 1 -- 172.21.15.79:0/492095081 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f093c004180 con 0x7f094c106ad0 2024-05-04T12:08:31.315 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:30.393+0000 7f094affd700 1 -- 172.21.15.79:0/492095081 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f093c0042e0 con 0x7f094c106ad0 2024-05-04T12:08:31.315 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:30.393+0000 7f094affd700 1 -- 172.21.15.79:0/492095081 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f093c011a50 con 0x7f094c106ad0 2024-05-04T12:08:31.315 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:30.393+0000 7f0952810700 1 -- 172.21.15.79:0/492095081 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f094c106ad0 msgr2=0x7f094c106ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:08:31.315 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:30.393+0000 7f0952810700 1 --2- 172.21.15.79:0/492095081 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f094c106ad0 0x7f094c106ef0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f093c0098a0 tx=0x7f093c009bb0 comp rx=0 tx=0).stop 2024-05-04T12:08:31.315 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:30.393+0000 7f0952810700 1 -- 172.21.15.79:0/492095081 shutdown_connections 2024-05-04T12:08:31.315 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:30.393+0000 7f0952810700 1 --2- 172.21.15.79:0/492095081 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f094c106ad0 0x7f094c106ef0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:08:31.316 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:30.393+0000 7f0952810700 1 -- 172.21.15.79:0/492095081 >> 172.21.15.79:0/492095081 conn(0x7f094c100190 msgr2=0x7f094c1025b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:08:31.316 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:30.393+0000 7f0952810700 1 -- 172.21.15.79:0/492095081 shutdown_connections 2024-05-04T12:08:31.316 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:30.394+0000 7f0952810700 1 -- 172.21.15.79:0/492095081 wait complete. 2024-05-04T12:08:31.316 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:30.394+0000 7f0952810700 1 Processor -- start 2024-05-04T12:08:31.316 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:30.394+0000 7f0952810700 1 -- start start 2024-05-04T12:08:31.316 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:30.394+0000 7f0952810700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f094c106ad0 0x7f094c190710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:08:31.316 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:30.394+0000 7f0952810700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f094c190c50 con 0x7f094c106ad0 2024-05-04T12:08:31.316 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:30.394+0000 7f094bfff700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f094c106ad0 0x7f094c190710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:08:31.316 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:30.395+0000 7f094bfff700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f094c106ad0 0x7f094c190710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:52464/0 (socket says 172.21.15.79:52464) 2024-05-04T12:08:31.316 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:30.395+0000 7f094bfff700 1 -- 172.21.15.79:0/3020922774 learned_addr learned my addr 172.21.15.79:0/3020922774 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:08:31.316 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:30.395+0000 7f094bfff700 1 -- 172.21.15.79:0/3020922774 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f093c009550 con 0x7f094c106ad0 2024-05-04T12:08:31.316 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:30.395+0000 7f094bfff700 1 --2- 172.21.15.79:0/3020922774 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f094c106ad0 0x7f094c190710 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f093c011a00 tx=0x7f093c011ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:08:31.316 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:30.395+0000 7f09497fa700 1 -- 172.21.15.79:0/3020922774 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f093c0038b0 con 0x7f094c106ad0 2024-05-04T12:08:31.316 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:30.395+0000 7f09497fa700 1 -- 172.21.15.79:0/3020922774 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f093c003a10 con 0x7f094c106ad0 2024-05-04T12:08:31.316 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:30.395+0000 7f09497fa700 1 -- 172.21.15.79:0/3020922774 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f093c0245a0 con 0x7f094c106ad0 2024-05-04T12:08:31.317 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:30.395+0000 7f0952810700 1 -- 172.21.15.79:0/3020922774 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f094c18ed60 con 0x7f094c106ad0 2024-05-04T12:08:31.317 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:30.395+0000 7f0952810700 1 -- 172.21.15.79:0/3020922774 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f094c18f200 con 0x7f094c106ad0 2024-05-04T12:08:31.317 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:30.396+0000 7f09497fa700 1 -- 172.21.15.79:0/3020922774 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7f093c02d3f0 con 0x7f094c106ad0 2024-05-04T12:08:31.317 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:30.396+0000 7f0952810700 1 -- 172.21.15.79:0/3020922774 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f094c0623c0 con 0x7f094c106ad0 2024-05-04T12:08:31.317 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:30.396+0000 7f09497fa700 1 --2- 172.21.15.79:0/3020922774 >> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] conn(0x7f0934038330 0x7f093403a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:08:31.317 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:30.396+0000 7f09497fa700 1 -- 172.21.15.79:0/3020922774 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f093c04c340 con 0x7f094c106ad0 2024-05-04T12:08:31.317 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:30.396+0000 7f094b7fe700 1 --2- 172.21.15.79:0/3020922774 >> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] conn(0x7f0934038330 0x7f093403a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:08:31.317 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:30.397+0000 7f094b7fe700 1 --2- 172.21.15.79:0/3020922774 >> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] conn(0x7f0934038330 0x7f093403a7f0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f0940006fd0 tx=0x7f0940006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:08:31.317 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:30.401+0000 7f09497fa700 1 -- 172.21.15.79:0/3020922774 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f093c0224f0 con 0x7f094c106ad0 2024-05-04T12:08:31.317 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:30.590+0000 7f0952810700 1 -- 172.21.15.79:0/3020922774 --> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f094c102320 con 0x7f0934038330 2024-05-04T12:08:31.317 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:30.796+0000 7f09497fa700 1 -- 172.21.15.79:0/3020922774 <== mgr.14120 v2:172.21.15.79:6800/1620705054 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f094c102320 con 0x7f0934038330 2024-05-04T12:08:31.317 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:30.798+0000 7f0952810700 1 -- 172.21.15.79:0/3020922774 >> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] conn(0x7f0934038330 msgr2=0x7f093403a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:08:31.317 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:30.798+0000 7f0952810700 1 --2- 172.21.15.79:0/3020922774 >> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] conn(0x7f0934038330 0x7f093403a7f0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f0940006fd0 tx=0x7f0940006e40 comp rx=0 tx=0).stop 2024-05-04T12:08:31.317 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:30.798+0000 7f0952810700 1 -- 172.21.15.79:0/3020922774 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f094c106ad0 msgr2=0x7f094c190710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:08:31.318 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:30.798+0000 7f0952810700 1 --2- 172.21.15.79:0/3020922774 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f094c106ad0 0x7f094c190710 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f093c011a00 tx=0x7f093c011ae0 comp rx=0 tx=0).stop 2024-05-04T12:08:31.318 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:30.798+0000 7f0952810700 1 -- 172.21.15.79:0/3020922774 shutdown_connections 2024-05-04T12:08:31.318 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:30.798+0000 7f0952810700 1 --2- 172.21.15.79:0/3020922774 >> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] conn(0x7f0934038330 0x7f093403a7f0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:08:31.318 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:30.798+0000 7f0952810700 1 --2- 172.21.15.79:0/3020922774 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f094c106ad0 0x7f094c190710 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:08:31.318 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:30.798+0000 7f0952810700 1 -- 172.21.15.79:0/3020922774 >> 172.21.15.79:0/3020922774 conn(0x7f094c100190 msgr2=0x7f094c104e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:08:31.318 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:30.799+0000 7f0952810700 1 -- 172.21.15.79:0/3020922774 shutdown_connections 2024-05-04T12:08:31.318 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:30.799+0000 7f0952810700 1 -- 172.21.15.79:0/3020922774 wait complete. 2024-05-04T12:08:32.195 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:08:31 smithi079 ceph-mon[27222]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-05-04T12:08:32.195 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:08:31 smithi079 ceph-mon[27222]: Generating ssh key... 2024-05-04T12:08:32.195 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:08:31 smithi079 ceph-mon[27222]: from='mgr.14120 172.21.15.79:0/2394226123' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:08:32.195 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:08:31 smithi079 ceph-mon[27222]: from='mgr.14120 172.21.15.79:0/2394226123' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:08:32.513 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCm7xcLdyPGUWO9LQ/mT1Bj8RF6DufxLVkza6YzpUq4GneHJDREQmAW/JWfepuD22rTR9AYtSiKp/G0PdhjHlVIf7A6NoWr3zuQFlBqe18A8sVglmLdRqex/E7LXYvPES34t8AmyLv+VOhRQ7gami3ArHe5RUV/C8XVjwksjitTyixV9Hl4V36A2vDWpkem49t0gMYY8OIOSRPBkq+P2KROawCZcONyfYApVbDL1548FIzbzfz3Lhlbok4PrOMs5TKkbw+/N+PT8NTB+wxpxN79gBYXoKQkt6MWUrKJLA2875Z6hXA1UF5MpY78P/fLjEtvMGCF0RRO7mv/I1EwGLao+AG/SkuVONFI8xE//aOKV4y/jm3xnimYSWawBXfnqeQNE+ZRs/T1NNP97jHmUWanqLDpbR6eDWXj2GY5heYL8w/r9ozypkns/G8yKpcy7fcGXwm0PI1scEikU+tw0G5G7nOxwGTqqMEhXgWNyah6q0z2rsaXTMxHyjxmVvxPYMU= ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968 2024-05-04T12:08:32.514 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:31.708+0000 7f802bd5f700 1 Processor -- start 2024-05-04T12:08:32.514 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:31.708+0000 7f802bd5f700 1 -- start start 2024-05-04T12:08:32.514 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:31.708+0000 7f802bd5f700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8024106560 0x7f8024106980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:08:32.514 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:31.708+0000 7f802bd5f700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8024106f50 con 0x7f8024106560 2024-05-04T12:08:32.514 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:31.708+0000 7f8029afb700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8024106560 0x7f8024106980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:08:32.514 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:31.708+0000 7f8029afb700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8024106560 0x7f8024106980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:52472/0 (socket says 172.21.15.79:52472) 2024-05-04T12:08:32.514 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:31.708+0000 7f8029afb700 1 -- 172.21.15.79:0/1941063041 learned_addr learned my addr 172.21.15.79:0/1941063041 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:08:32.514 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:31.709+0000 7f8029afb700 1 -- 172.21.15.79:0/1941063041 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8024107760 con 0x7f8024106560 2024-05-04T12:08:32.514 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:31.709+0000 7f8029afb700 1 --2- 172.21.15.79:0/1941063041 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8024106560 0x7f8024106980 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f80200098a0 tx=0x7f8020009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b7c51156f6ec4be server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:08:32.514 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:31.709+0000 7f8028af9700 1 -- 172.21.15.79:0/1941063041 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8020004180 con 0x7f8024106560 2024-05-04T12:08:32.514 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:31.709+0000 7f8028af9700 1 -- 172.21.15.79:0/1941063041 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f80200042e0 con 0x7f8024106560 2024-05-04T12:08:32.514 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:31.709+0000 7f8028af9700 1 -- 172.21.15.79:0/1941063041 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8020011a50 con 0x7f8024106560 2024-05-04T12:08:32.515 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:31.709+0000 7f802bd5f700 1 -- 172.21.15.79:0/1941063041 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8024106560 msgr2=0x7f8024106980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:08:32.515 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:31.709+0000 7f802bd5f700 1 --2- 172.21.15.79:0/1941063041 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8024106560 0x7f8024106980 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f80200098a0 tx=0x7f8020009bb0 comp rx=0 tx=0).stop 2024-05-04T12:08:32.515 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:31.710+0000 7f802bd5f700 1 -- 172.21.15.79:0/1941063041 shutdown_connections 2024-05-04T12:08:32.515 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:31.710+0000 7f802bd5f700 1 --2- 172.21.15.79:0/1941063041 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8024106560 0x7f8024106980 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:08:32.515 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:31.710+0000 7f802bd5f700 1 -- 172.21.15.79:0/1941063041 >> 172.21.15.79:0/1941063041 conn(0x7f8024101b40 msgr2=0x7f8024103f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:08:32.515 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:31.710+0000 7f802bd5f700 1 -- 172.21.15.79:0/1941063041 shutdown_connections 2024-05-04T12:08:32.515 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:31.710+0000 7f802bd5f700 1 -- 172.21.15.79:0/1941063041 wait complete. 2024-05-04T12:08:32.515 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:31.710+0000 7f802bd5f700 1 Processor -- start 2024-05-04T12:08:32.515 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:31.710+0000 7f802bd5f700 1 -- start start 2024-05-04T12:08:32.515 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:31.710+0000 7f802bd5f700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8024106560 0x7f802419a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:08:32.515 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:31.710+0000 7f802bd5f700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f802419a730 con 0x7f8024106560 2024-05-04T12:08:32.515 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:31.710+0000 7f8029afb700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8024106560 0x7f802419a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:08:32.515 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:31.711+0000 7f8029afb700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8024106560 0x7f802419a1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:52474/0 (socket says 172.21.15.79:52474) 2024-05-04T12:08:32.515 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:31.711+0000 7f8029afb700 1 -- 172.21.15.79:0/2065973892 learned_addr learned my addr 172.21.15.79:0/2065973892 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:08:32.516 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:31.711+0000 7f8029afb700 1 -- 172.21.15.79:0/2065973892 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8020009550 con 0x7f8024106560 2024-05-04T12:08:32.516 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:31.711+0000 7f8029afb700 1 --2- 172.21.15.79:0/2065973892 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8024106560 0x7f802419a1f0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f8020009520 tx=0x7f8020011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:08:32.516 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:31.711+0000 7f801affd700 1 -- 172.21.15.79:0/2065973892 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8020003a80 con 0x7f8024106560 2024-05-04T12:08:32.516 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:31.711+0000 7f801affd700 1 -- 172.21.15.79:0/2065973892 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f8020003be0 con 0x7f8024106560 2024-05-04T12:08:32.516 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:31.711+0000 7f802bd5f700 1 -- 172.21.15.79:0/2065973892 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f802419a930 con 0x7f8024106560 2024-05-04T12:08:32.516 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:31.711+0000 7f801affd700 1 -- 172.21.15.79:0/2065973892 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f80200197b0 con 0x7f8024106560 2024-05-04T12:08:32.516 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:31.711+0000 7f802bd5f700 1 -- 172.21.15.79:0/2065973892 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f802419add0 con 0x7f8024106560 2024-05-04T12:08:32.516 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:31.712+0000 7f801affd700 1 -- 172.21.15.79:0/2065973892 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7f8020019a70 con 0x7f8024106560 2024-05-04T12:08:32.516 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:31.712+0000 7f802bd5f700 1 -- 172.21.15.79:0/2065973892 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80240623c0 con 0x7f8024106560 2024-05-04T12:08:32.516 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:31.712+0000 7f801affd700 1 --2- 172.21.15.79:0/2065973892 >> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] conn(0x7f80100382d0 0x7f801003a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:08:32.516 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:31.712+0000 7f801affd700 1 -- 172.21.15.79:0/2065973892 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f802004c5c0 con 0x7f8024106560 2024-05-04T12:08:32.516 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:31.712+0000 7f80292fa700 1 --2- 172.21.15.79:0/2065973892 >> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] conn(0x7f80100382d0 0x7f801003a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:08:32.516 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:31.713+0000 7f80292fa700 1 --2- 172.21.15.79:0/2065973892 >> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] conn(0x7f80100382d0 0x7f801003a790 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f8014006fd0 tx=0x7f8014006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:08:32.517 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:31.717+0000 7f801affd700 1 -- 172.21.15.79:0/2065973892 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f802001ac30 con 0x7f8024106560 2024-05-04T12:08:32.517 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:31.908+0000 7f802bd5f700 1 -- 172.21.15.79:0/2065973892 --> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f802410a960 con 0x7f80100382d0 2024-05-04T12:08:32.517 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:31.908+0000 7f801affd700 1 -- 172.21.15.79:0/2065973892 <== mgr.14120 v2:172.21.15.79:6800/1620705054 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7f802410a960 con 0x7f80100382d0 2024-05-04T12:08:32.517 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:31.910+0000 7f802bd5f700 1 -- 172.21.15.79:0/2065973892 >> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] conn(0x7f80100382d0 msgr2=0x7f801003a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:08:32.517 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:31.910+0000 7f802bd5f700 1 --2- 172.21.15.79:0/2065973892 >> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] conn(0x7f80100382d0 0x7f801003a790 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f8014006fd0 tx=0x7f8014006e40 comp rx=0 tx=0).stop 2024-05-04T12:08:32.517 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:31.910+0000 7f802bd5f700 1 -- 172.21.15.79:0/2065973892 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8024106560 msgr2=0x7f802419a1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:08:32.517 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:31.910+0000 7f802bd5f700 1 --2- 172.21.15.79:0/2065973892 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8024106560 0x7f802419a1f0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f8020009520 tx=0x7f8020011ab0 comp rx=0 tx=0).stop 2024-05-04T12:08:32.517 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:31.910+0000 7f802bd5f700 1 -- 172.21.15.79:0/2065973892 shutdown_connections 2024-05-04T12:08:32.517 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:31.910+0000 7f802bd5f700 1 --2- 172.21.15.79:0/2065973892 >> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] conn(0x7f80100382d0 0x7f801003a790 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:08:32.517 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:31.910+0000 7f802bd5f700 1 --2- 172.21.15.79:0/2065973892 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8024106560 0x7f802419a1f0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:08:32.517 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:31.910+0000 7f802bd5f700 1 -- 172.21.15.79:0/2065973892 >> 172.21.15.79:0/2065973892 conn(0x7f8024101b40 msgr2=0x7f8024102900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:08:32.517 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:31.911+0000 7f802bd5f700 1 -- 172.21.15.79:0/2065973892 shutdown_connections 2024-05-04T12:08:32.517 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:31.911+0000 7f802bd5f700 1 -- 172.21.15.79:0/2065973892 wait complete. 2024-05-04T12:08:32.517 INFO:teuthology.orchestra.run.smithi079.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-05-04T12:08:32.518 INFO:teuthology.orchestra.run.smithi079.stdout:Adding key to root@localhost authorized_keys... 2024-05-04T12:08:32.518 INFO:teuthology.orchestra.run.smithi079.stdout:Adding host smithi079... 2024-05-04T12:08:33.080 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:08:32 smithi079 ceph-mon[27222]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-05-04T12:08:33.945 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:08:33 smithi079 ceph-mon[27222]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi079", "addr": "172.21.15.79", "target": ["mon-mgr", ""]}]: dispatch 2024-05-04T12:08:35.069 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:08:34 smithi079 ceph-mon[27222]: Deploying cephadm binary to smithi079 2024-05-04T12:08:36.172 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout Added host 'smithi079' with addr '172.21.15.79' 2024-05-04T12:08:36.172 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:32.996+0000 7fc9a98c9700 1 Processor -- start 2024-05-04T12:08:36.172 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:32.996+0000 7fc9a98c9700 1 -- start start 2024-05-04T12:08:36.172 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:32.996+0000 7fc9a98c9700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc9a4105460 0x7fc9a4105880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:08:36.173 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:32.996+0000 7fc9a98c9700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9a4105e50 con 0x7fc9a4105460 2024-05-04T12:08:36.173 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:32.996+0000 7fc9a2ffd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc9a4105460 0x7fc9a4105880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:08:36.173 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:32.996+0000 7fc9a2ffd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc9a4105460 0x7fc9a4105880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:52476/0 (socket says 172.21.15.79:52476) 2024-05-04T12:08:36.173 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:32.996+0000 7fc9a2ffd700 1 -- 172.21.15.79:0/1277339072 learned_addr learned my addr 172.21.15.79:0/1277339072 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:08:36.173 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:32.996+0000 7fc9a2ffd700 1 -- 172.21.15.79:0/1277339072 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9a4106660 con 0x7fc9a4105460 2024-05-04T12:08:36.173 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:32.997+0000 7fc9a2ffd700 1 --2- 172.21.15.79:0/1277339072 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc9a4105460 0x7fc9a4105880 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fc9940098a0 tx=0x7fc994009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5cabe90541f091bf server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:08:36.173 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:32.997+0000 7fc9a1ffb700 1 -- 172.21.15.79:0/1277339072 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc994004180 con 0x7fc9a4105460 2024-05-04T12:08:36.173 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:32.997+0000 7fc9a1ffb700 1 -- 172.21.15.79:0/1277339072 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fc9940042e0 con 0x7fc9a4105460 2024-05-04T12:08:36.173 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:32.997+0000 7fc9a1ffb700 1 -- 172.21.15.79:0/1277339072 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc994011a50 con 0x7fc9a4105460 2024-05-04T12:08:36.173 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:32.997+0000 7fc9a98c9700 1 -- 172.21.15.79:0/1277339072 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc9a4105460 msgr2=0x7fc9a4105880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:08:36.173 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:32.997+0000 7fc9a98c9700 1 --2- 172.21.15.79:0/1277339072 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc9a4105460 0x7fc9a4105880 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fc9940098a0 tx=0x7fc994009bb0 comp rx=0 tx=0).stop 2024-05-04T12:08:36.173 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:32.997+0000 7fc9a98c9700 1 -- 172.21.15.79:0/1277339072 shutdown_connections 2024-05-04T12:08:36.173 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:32.997+0000 7fc9a98c9700 1 --2- 172.21.15.79:0/1277339072 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc9a4105460 0x7fc9a4105880 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:08:36.174 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:32.997+0000 7fc9a98c9700 1 -- 172.21.15.79:0/1277339072 >> 172.21.15.79:0/1277339072 conn(0x7fc9a4100a20 msgr2=0x7fc9a4102e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:08:36.174 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:32.998+0000 7fc9a98c9700 1 -- 172.21.15.79:0/1277339072 shutdown_connections 2024-05-04T12:08:36.174 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:32.998+0000 7fc9a98c9700 1 -- 172.21.15.79:0/1277339072 wait complete. 2024-05-04T12:08:36.174 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:32.998+0000 7fc9a98c9700 1 Processor -- start 2024-05-04T12:08:36.174 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:32.998+0000 7fc9a98c9700 1 -- start start 2024-05-04T12:08:36.174 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:32.998+0000 7fc9a98c9700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc9a4105460 0x7fc9a419a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:08:36.174 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:32.998+0000 7fc9a98c9700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9a419a750 con 0x7fc9a4105460 2024-05-04T12:08:36.174 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:32.999+0000 7fc9a2ffd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc9a4105460 0x7fc9a419a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:08:36.174 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:32.999+0000 7fc9a2ffd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc9a4105460 0x7fc9a419a210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:52488/0 (socket says 172.21.15.79:52488) 2024-05-04T12:08:36.174 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:32.999+0000 7fc9a2ffd700 1 -- 172.21.15.79:0/2781548440 learned_addr learned my addr 172.21.15.79:0/2781548440 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:08:36.174 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:32.999+0000 7fc9a2ffd700 1 -- 172.21.15.79:0/2781548440 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc994009550 con 0x7fc9a4105460 2024-05-04T12:08:36.174 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:32.999+0000 7fc9a2ffd700 1 --2- 172.21.15.79:0/2781548440 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc9a4105460 0x7fc9a419a210 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fc994009520 tx=0x7fc994011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:08:36.174 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:32.999+0000 7fc9a88c7700 1 -- 172.21.15.79:0/2781548440 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc994003a80 con 0x7fc9a4105460 2024-05-04T12:08:36.175 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:32.999+0000 7fc9a88c7700 1 -- 172.21.15.79:0/2781548440 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fc994003be0 con 0x7fc9a4105460 2024-05-04T12:08:36.175 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:32.999+0000 7fc9a88c7700 1 -- 172.21.15.79:0/2781548440 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc9940197b0 con 0x7fc9a4105460 2024-05-04T12:08:36.175 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:32.999+0000 7fc9a98c9700 1 -- 172.21.15.79:0/2781548440 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9a419a950 con 0x7fc9a4105460 2024-05-04T12:08:36.175 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:32.999+0000 7fc9a98c9700 1 -- 172.21.15.79:0/2781548440 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9a419adf0 con 0x7fc9a4105460 2024-05-04T12:08:36.175 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:33.000+0000 7fc9a88c7700 1 -- 172.21.15.79:0/2781548440 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7fc994019a20 con 0x7fc9a4105460 2024-05-04T12:08:36.175 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:33.000+0000 7fc9a88c7700 1 --2- 172.21.15.79:0/2781548440 >> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] conn(0x7fc98c038320 0x7fc98c03a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:08:36.175 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:33.000+0000 7fc9a88c7700 1 -- 172.21.15.79:0/2781548440 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fc99404c540 con 0x7fc9a4105460 2024-05-04T12:08:36.175 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:33.000+0000 7fc9a27fc700 1 --2- 172.21.15.79:0/2781548440 >> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] conn(0x7fc98c038320 0x7fc98c03a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:08:36.175 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:33.000+0000 7fc9a98c9700 1 -- 172.21.15.79:0/2781548440 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9a40623c0 con 0x7fc9a4105460 2024-05-04T12:08:36.175 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:33.001+0000 7fc9a27fc700 1 --2- 172.21.15.79:0/2781548440 >> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] conn(0x7fc98c038320 0x7fc98c03a7e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fc998006fd0 tx=0x7fc998006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:08:36.175 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:33.005+0000 7fc9a88c7700 1 -- 172.21.15.79:0/2781548440 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fc99402c3a0 con 0x7fc9a4105460 2024-05-04T12:08:36.175 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:33.229+0000 7fc9a98c9700 1 -- 172.21.15.79:0/2781548440 --> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi079", "addr": "172.21.15.79", "target": ["mon-mgr", ""]}) v1 -- 0x7fc9a4102ba0 con 0x7fc98c038320 2024-05-04T12:08:36.176 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:35.585+0000 7fc9a88c7700 1 -- 172.21.15.79:0/2781548440 <== mgr.14120 v2:172.21.15.79:6800/1620705054 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7fc9a4102ba0 con 0x7fc98c038320 2024-05-04T12:08:36.176 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:35.587+0000 7fc9a98c9700 1 -- 172.21.15.79:0/2781548440 >> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] conn(0x7fc98c038320 msgr2=0x7fc98c03a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:08:36.176 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:35.587+0000 7fc9a98c9700 1 --2- 172.21.15.79:0/2781548440 >> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] conn(0x7fc98c038320 0x7fc98c03a7e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fc998006fd0 tx=0x7fc998006e40 comp rx=0 tx=0).stop 2024-05-04T12:08:36.176 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:35.587+0000 7fc9a98c9700 1 -- 172.21.15.79:0/2781548440 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc9a4105460 msgr2=0x7fc9a419a210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:08:36.176 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:35.587+0000 7fc9a98c9700 1 --2- 172.21.15.79:0/2781548440 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc9a4105460 0x7fc9a419a210 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fc994009520 tx=0x7fc994011ab0 comp rx=0 tx=0).stop 2024-05-04T12:08:36.176 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:35.588+0000 7fc9a98c9700 1 -- 172.21.15.79:0/2781548440 shutdown_connections 2024-05-04T12:08:36.176 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:35.588+0000 7fc9a98c9700 1 --2- 172.21.15.79:0/2781548440 >> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] conn(0x7fc98c038320 0x7fc98c03a7e0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:08:36.176 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:35.588+0000 7fc9a98c9700 1 --2- 172.21.15.79:0/2781548440 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc9a4105460 0x7fc9a419a210 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:08:36.176 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:35.588+0000 7fc9a98c9700 1 -- 172.21.15.79:0/2781548440 >> 172.21.15.79:0/2781548440 conn(0x7fc9a4100a20 msgr2=0x7fc9a4079840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:08:36.176 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:35.588+0000 7fc9a98c9700 1 -- 172.21.15.79:0/2781548440 shutdown_connections 2024-05-04T12:08:36.176 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:35.588+0000 7fc9a98c9700 1 -- 172.21.15.79:0/2781548440 wait complete. 2024-05-04T12:08:36.176 INFO:teuthology.orchestra.run.smithi079.stdout:Deploying mon service with default placement... 2024-05-04T12:08:36.787 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:08:36 smithi079 ceph-mon[27222]: from='mgr.14120 172.21.15.79:0/2394226123' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:08:36.787 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:08:36 smithi079 ceph-mon[27222]: Added host smithi079 2024-05-04T12:08:36.787 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:08:36 smithi079 ceph-mon[27222]: from='mgr.14120 172.21.15.79:0/2394226123' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-04T12:08:37.422 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-05-04T12:08:37.422 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:36.569+0000 7f504d661700 1 Processor -- start 2024-05-04T12:08:37.422 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:36.569+0000 7f504d661700 1 -- start start 2024-05-04T12:08:37.422 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:36.569+0000 7f504d661700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5048106560 0x7f5048106980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:08:37.423 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:36.569+0000 7f504d661700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5048106f50 con 0x7f5048106560 2024-05-04T12:08:37.423 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:36.569+0000 7f5046ffd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5048106560 0x7f5048106980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:08:37.423 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:36.569+0000 7f5046ffd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5048106560 0x7f5048106980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:57204/0 (socket says 172.21.15.79:57204) 2024-05-04T12:08:37.423 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:36.569+0000 7f5046ffd700 1 -- 172.21.15.79:0/1270888858 learned_addr learned my addr 172.21.15.79:0/1270888858 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:08:37.423 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:36.569+0000 7f5046ffd700 1 -- 172.21.15.79:0/1270888858 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5048107760 con 0x7f5048106560 2024-05-04T12:08:37.423 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:36.570+0000 7f5046ffd700 1 --2- 172.21.15.79:0/1270888858 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5048106560 0x7f5048106980 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f50380098a0 tx=0x7f5038009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=806edf795c69d342 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:08:37.423 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:36.570+0000 7f5045ffb700 1 -- 172.21.15.79:0/1270888858 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5038004180 con 0x7f5048106560 2024-05-04T12:08:37.423 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:36.570+0000 7f5045ffb700 1 -- 172.21.15.79:0/1270888858 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f50380042e0 con 0x7f5048106560 2024-05-04T12:08:37.423 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:36.570+0000 7f5045ffb700 1 -- 172.21.15.79:0/1270888858 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5038011a50 con 0x7f5048106560 2024-05-04T12:08:37.423 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:36.570+0000 7f504d661700 1 -- 172.21.15.79:0/1270888858 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5048106560 msgr2=0x7f5048106980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:08:37.423 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:36.570+0000 7f504d661700 1 --2- 172.21.15.79:0/1270888858 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5048106560 0x7f5048106980 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f50380098a0 tx=0x7f5038009bb0 comp rx=0 tx=0).stop 2024-05-04T12:08:37.423 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:36.570+0000 7f504d661700 1 -- 172.21.15.79:0/1270888858 shutdown_connections 2024-05-04T12:08:37.423 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:36.570+0000 7f504d661700 1 --2- 172.21.15.79:0/1270888858 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5048106560 0x7f5048106980 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:08:37.423 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:36.570+0000 7f504d661700 1 -- 172.21.15.79:0/1270888858 >> 172.21.15.79:0/1270888858 conn(0x7f5048101b40 msgr2=0x7f5048103f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:08:37.424 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:36.570+0000 7f504d661700 1 -- 172.21.15.79:0/1270888858 shutdown_connections 2024-05-04T12:08:37.424 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:36.570+0000 7f504d661700 1 -- 172.21.15.79:0/1270888858 wait complete. 2024-05-04T12:08:37.424 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:36.571+0000 7f504d661700 1 Processor -- start 2024-05-04T12:08:37.424 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:36.571+0000 7f504d661700 1 -- start start 2024-05-04T12:08:37.424 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:36.571+0000 7f504d661700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5048106560 0x7f504819c380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:08:37.424 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:36.571+0000 7f504d661700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f504819c8c0 con 0x7f5048106560 2024-05-04T12:08:37.424 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:36.571+0000 7f5046ffd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5048106560 0x7f504819c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:08:37.424 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:36.571+0000 7f5046ffd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5048106560 0x7f504819c380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:57206/0 (socket says 172.21.15.79:57206) 2024-05-04T12:08:37.424 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:36.571+0000 7f5046ffd700 1 -- 172.21.15.79:0/1576103917 learned_addr learned my addr 172.21.15.79:0/1576103917 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:08:37.424 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:36.572+0000 7f5046ffd700 1 -- 172.21.15.79:0/1576103917 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5038009550 con 0x7f5048106560 2024-05-04T12:08:37.424 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:36.572+0000 7f5046ffd700 1 --2- 172.21.15.79:0/1576103917 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5048106560 0x7f504819c380 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f5038003de0 tx=0x7f5038011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:08:37.424 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:36.572+0000 7f502ffff700 1 -- 172.21.15.79:0/1576103917 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5038003930 con 0x7f5048106560 2024-05-04T12:08:37.424 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:36.572+0000 7f502ffff700 1 -- 172.21.15.79:0/1576103917 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f5038003a90 con 0x7f5048106560 2024-05-04T12:08:37.424 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:36.572+0000 7f504d661700 1 -- 172.21.15.79:0/1576103917 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f504819cac0 con 0x7f5048106560 2024-05-04T12:08:37.425 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:36.572+0000 7f502ffff700 1 -- 172.21.15.79:0/1576103917 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f50380196c0 con 0x7f5048106560 2024-05-04T12:08:37.425 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:36.572+0000 7f504d661700 1 -- 172.21.15.79:0/1576103917 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f504819cf60 con 0x7f5048106560 2024-05-04T12:08:37.425 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:36.573+0000 7f502ffff700 1 -- 172.21.15.79:0/1576103917 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7f5038019a00 con 0x7f5048106560 2024-05-04T12:08:37.425 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:36.573+0000 7f504d661700 1 -- 172.21.15.79:0/1576103917 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50480623c0 con 0x7f5048106560 2024-05-04T12:08:37.425 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:36.573+0000 7f502ffff700 1 --2- 172.21.15.79:0/1576103917 >> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] conn(0x7f5030038320 0x7f503003a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:08:37.425 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:36.573+0000 7f502ffff700 1 -- 172.21.15.79:0/1576103917 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f503804c440 con 0x7f5048106560 2024-05-04T12:08:37.425 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:36.573+0000 7f50467fc700 1 --2- 172.21.15.79:0/1576103917 >> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] conn(0x7f5030038320 0x7f503003a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:08:37.425 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:36.574+0000 7f50467fc700 1 --2- 172.21.15.79:0/1576103917 >> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] conn(0x7f5030038320 0x7f503003a7e0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f503c006fd0 tx=0x7f503c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:08:37.425 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:36.578+0000 7f502ffff700 1 -- 172.21.15.79:0/1576103917 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f5038023370 con 0x7f5048106560 2024-05-04T12:08:37.425 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:36.787+0000 7f504d661700 1 -- 172.21.15.79:0/1576103917 --> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}) v1 -- 0x7f504810a960 con 0x7f5030038320 2024-05-04T12:08:37.425 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:36.792+0000 7f502ffff700 1 -- 172.21.15.79:0/1576103917 <== mgr.14120 v2:172.21.15.79:6800/1620705054 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f504810a960 con 0x7f5030038320 2024-05-04T12:08:37.425 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:36.794+0000 7f504d661700 1 -- 172.21.15.79:0/1576103917 >> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] conn(0x7f5030038320 msgr2=0x7f503003a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:08:37.425 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:36.794+0000 7f504d661700 1 --2- 172.21.15.79:0/1576103917 >> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] conn(0x7f5030038320 0x7f503003a7e0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f503c006fd0 tx=0x7f503c006e40 comp rx=0 tx=0).stop 2024-05-04T12:08:37.425 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:36.794+0000 7f504d661700 1 -- 172.21.15.79:0/1576103917 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5048106560 msgr2=0x7f504819c380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:08:37.426 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:36.794+0000 7f504d661700 1 --2- 172.21.15.79:0/1576103917 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5048106560 0x7f504819c380 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f5038003de0 tx=0x7f5038011ab0 comp rx=0 tx=0).stop 2024-05-04T12:08:37.426 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:36.794+0000 7f504d661700 1 -- 172.21.15.79:0/1576103917 shutdown_connections 2024-05-04T12:08:37.426 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:36.794+0000 7f504d661700 1 --2- 172.21.15.79:0/1576103917 >> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] conn(0x7f5030038320 0x7f503003a7e0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:08:37.426 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:36.794+0000 7f504d661700 1 --2- 172.21.15.79:0/1576103917 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5048106560 0x7f504819c380 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:08:37.426 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:36.794+0000 7f504d661700 1 -- 172.21.15.79:0/1576103917 >> 172.21.15.79:0/1576103917 conn(0x7f5048101b40 msgr2=0x7f50481028c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:08:37.426 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:36.795+0000 7f504d661700 1 -- 172.21.15.79:0/1576103917 shutdown_connections 2024-05-04T12:08:37.426 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:36.795+0000 7f504d661700 1 -- 172.21.15.79:0/1576103917 wait complete. 2024-05-04T12:08:37.426 INFO:teuthology.orchestra.run.smithi079.stdout:Deploying mgr service with default placement... 2024-05-04T12:08:38.135 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:08:37 smithi079 ceph-mon[27222]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2024-05-04T12:08:38.135 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:08:37 smithi079 ceph-mon[27222]: Saving service mon spec with placement count:5 2024-05-04T12:08:38.135 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:08:37 smithi079 ceph-mon[27222]: from='mgr.14120 172.21.15.79:0/2394226123' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:08:39.409 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:08:39 smithi079 ceph-mon[27222]: from='mgr.14120 172.21.15.79:0/2394226123' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:08:39.409 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:08:39 smithi079 ceph-mon[27222]: from='mgr.14120 172.21.15.79:0/2394226123' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:08:39.409 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:08:39 smithi079 ceph-mon[27222]: from='mgr.14120 172.21.15.79:0/2394226123' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:08:39.417 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-05-04T12:08:39.417 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:38.567+0000 7f1dfae34700 1 Processor -- start 2024-05-04T12:08:39.417 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:38.567+0000 7f1dfae34700 1 -- start start 2024-05-04T12:08:39.417 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:38.567+0000 7f1dfae34700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1df4106a60 0x7f1df4106e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:08:39.417 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:38.567+0000 7f1dfae34700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1df4107450 con 0x7f1df4106a60 2024-05-04T12:08:39.418 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:38.567+0000 7f1df8bd0700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1df4106a60 0x7f1df4106e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:08:39.418 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:38.567+0000 7f1df8bd0700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1df4106a60 0x7f1df4106e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:57222/0 (socket says 172.21.15.79:57222) 2024-05-04T12:08:39.418 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:38.567+0000 7f1df8bd0700 1 -- 172.21.15.79:0/3683127395 learned_addr learned my addr 172.21.15.79:0/3683127395 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:08:39.418 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:38.568+0000 7f1df8bd0700 1 -- 172.21.15.79:0/3683127395 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1df41056c0 con 0x7f1df4106a60 2024-05-04T12:08:39.418 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:38.568+0000 7f1df8bd0700 1 --2- 172.21.15.79:0/3683127395 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1df4106a60 0x7f1df4106e80 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f1de80098a0 tx=0x7f1de8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b264c32721248c9 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:08:39.418 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:38.568+0000 7f1df37fe700 1 -- 172.21.15.79:0/3683127395 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1de8004180 con 0x7f1df4106a60 2024-05-04T12:08:39.418 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:38.568+0000 7f1df37fe700 1 -- 172.21.15.79:0/3683127395 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f1de80042e0 con 0x7f1df4106a60 2024-05-04T12:08:39.418 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:38.568+0000 7f1df37fe700 1 -- 172.21.15.79:0/3683127395 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1de8011af0 con 0x7f1df4106a60 2024-05-04T12:08:39.418 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:38.569+0000 7f1dfae34700 1 -- 172.21.15.79:0/3683127395 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1df4106a60 msgr2=0x7f1df4106e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:08:39.418 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:38.569+0000 7f1dfae34700 1 --2- 172.21.15.79:0/3683127395 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1df4106a60 0x7f1df4106e80 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f1de80098a0 tx=0x7f1de8009bb0 comp rx=0 tx=0).stop 2024-05-04T12:08:39.418 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:38.569+0000 7f1dfae34700 1 -- 172.21.15.79:0/3683127395 shutdown_connections 2024-05-04T12:08:39.418 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:38.569+0000 7f1dfae34700 1 --2- 172.21.15.79:0/3683127395 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1df4106a60 0x7f1df4106e80 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:08:39.418 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:38.569+0000 7f1dfae34700 1 -- 172.21.15.79:0/3683127395 >> 172.21.15.79:0/3683127395 conn(0x7f1df41001d0 msgr2=0x7f1df41025f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:08:39.419 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:38.569+0000 7f1dfae34700 1 -- 172.21.15.79:0/3683127395 shutdown_connections 2024-05-04T12:08:39.419 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:38.569+0000 7f1dfae34700 1 -- 172.21.15.79:0/3683127395 wait complete. 2024-05-04T12:08:39.419 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:38.569+0000 7f1dfae34700 1 Processor -- start 2024-05-04T12:08:39.419 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:38.569+0000 7f1dfae34700 1 -- start start 2024-05-04T12:08:39.419 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:38.569+0000 7f1dfae34700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1df4106a60 0x7f1df4105170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:08:39.419 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:38.569+0000 7f1dfae34700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1df419cf60 con 0x7f1df4106a60 2024-05-04T12:08:39.419 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:38.569+0000 7f1df8bd0700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1df4106a60 0x7f1df4105170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:08:39.419 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:38.570+0000 7f1df8bd0700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1df4106a60 0x7f1df4105170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:57226/0 (socket says 172.21.15.79:57226) 2024-05-04T12:08:39.419 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:38.570+0000 7f1df8bd0700 1 -- 172.21.15.79:0/1444547554 learned_addr learned my addr 172.21.15.79:0/1444547554 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:08:39.419 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:38.570+0000 7f1df8bd0700 1 -- 172.21.15.79:0/1444547554 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1de8009550 con 0x7f1df4106a60 2024-05-04T12:08:39.419 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:38.570+0000 7f1df8bd0700 1 --2- 172.21.15.79:0/1444547554 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1df4106a60 0x7f1df4105170 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f1de8011670 tx=0x7f1de8011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:08:39.419 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:38.570+0000 7f1df1ffb700 1 -- 172.21.15.79:0/1444547554 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1de80039f0 con 0x7f1df4106a60 2024-05-04T12:08:39.419 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:38.570+0000 7f1dfae34700 1 -- 172.21.15.79:0/1444547554 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1df419d0a0 con 0x7f1df4106a60 2024-05-04T12:08:39.420 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:38.570+0000 7f1df1ffb700 1 -- 172.21.15.79:0/1444547554 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f1de8003b50 con 0x7f1df4106a60 2024-05-04T12:08:39.420 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:38.570+0000 7f1df1ffb700 1 -- 172.21.15.79:0/1444547554 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1de80247c0 con 0x7f1df4106a60 2024-05-04T12:08:39.420 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:38.570+0000 7f1dfae34700 1 -- 172.21.15.79:0/1444547554 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1df419d540 con 0x7f1df4106a60 2024-05-04T12:08:39.420 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:38.571+0000 7f1df1ffb700 1 -- 172.21.15.79:0/1444547554 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7f1de801a430 con 0x7f1df4106a60 2024-05-04T12:08:39.420 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:38.571+0000 7f1df1ffb700 1 --2- 172.21.15.79:0/1444547554 >> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] conn(0x7f1de4038370 0x7f1de403a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:08:39.420 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:38.571+0000 7f1dfae34700 1 -- 172.21.15.79:0/1444547554 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1df40623c0 con 0x7f1df4106a60 2024-05-04T12:08:39.420 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:38.571+0000 7f1df1ffb700 1 -- 172.21.15.79:0/1444547554 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f1de804c370 con 0x7f1df4106a60 2024-05-04T12:08:39.420 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:38.571+0000 7f1df3fff700 1 --2- 172.21.15.79:0/1444547554 >> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] conn(0x7f1de4038370 0x7f1de403a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:08:39.420 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:38.572+0000 7f1df3fff700 1 --2- 172.21.15.79:0/1444547554 >> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] conn(0x7f1de4038370 0x7f1de403a830 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f1de0006fd0 tx=0x7f1de0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:08:39.420 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:38.576+0000 7f1df1ffb700 1 -- 172.21.15.79:0/1444547554 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f1de8018b40 con 0x7f1df4106a60 2024-05-04T12:08:39.420 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:38.792+0000 7f1dfae34700 1 -- 172.21.15.79:0/1444547554 --> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7f1df4101680 con 0x7f1de4038370 2024-05-04T12:08:39.420 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:38.800+0000 7f1df1ffb700 1 -- 172.21.15.79:0/1444547554 <== mgr.14120 v2:172.21.15.79:6800/1620705054 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f1df4101680 con 0x7f1de4038370 2024-05-04T12:08:39.420 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:38.802+0000 7f1dfae34700 1 -- 172.21.15.79:0/1444547554 >> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] conn(0x7f1de4038370 msgr2=0x7f1de403a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:08:39.420 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:38.802+0000 7f1dfae34700 1 --2- 172.21.15.79:0/1444547554 >> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] conn(0x7f1de4038370 0x7f1de403a830 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f1de0006fd0 tx=0x7f1de0006e40 comp rx=0 tx=0).stop 2024-05-04T12:08:39.421 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:38.802+0000 7f1dfae34700 1 -- 172.21.15.79:0/1444547554 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1df4106a60 msgr2=0x7f1df4105170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:08:39.421 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:38.802+0000 7f1dfae34700 1 --2- 172.21.15.79:0/1444547554 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1df4106a60 0x7f1df4105170 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f1de8011670 tx=0x7f1de8011b50 comp rx=0 tx=0).stop 2024-05-04T12:08:39.421 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:38.802+0000 7f1dfae34700 1 -- 172.21.15.79:0/1444547554 shutdown_connections 2024-05-04T12:08:39.421 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:38.802+0000 7f1dfae34700 1 --2- 172.21.15.79:0/1444547554 >> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] conn(0x7f1de4038370 0x7f1de403a830 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:08:39.421 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:38.802+0000 7f1dfae34700 1 --2- 172.21.15.79:0/1444547554 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1df4106a60 0x7f1df4105170 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:08:39.421 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:38.802+0000 7f1dfae34700 1 -- 172.21.15.79:0/1444547554 >> 172.21.15.79:0/1444547554 conn(0x7f1df41001d0 msgr2=0x7f1df4100f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:08:39.421 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:38.802+0000 7f1dfae34700 1 -- 172.21.15.79:0/1444547554 shutdown_connections 2024-05-04T12:08:39.421 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:38.802+0000 7f1dfae34700 1 -- 172.21.15.79:0/1444547554 wait complete. 2024-05-04T12:08:39.421 INFO:teuthology.orchestra.run.smithi079.stdout:Deploying crash service with default placement... 2024-05-04T12:08:40.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:08:40 smithi079 ceph-mon[27222]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-05-04T12:08:40.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:08:40 smithi079 ceph-mon[27222]: Saving service mgr spec with placement count:2 2024-05-04T12:08:40.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:08:40 smithi079 ceph-mon[27222]: from='mgr.14120 172.21.15.79:0/2394226123' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:08:40.809 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout Scheduled crash update... 2024-05-04T12:08:40.809 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:39.810+0000 7f1eafe95700 1 Processor -- start 2024-05-04T12:08:40.809 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:39.810+0000 7f1eafe95700 1 -- start start 2024-05-04T12:08:40.809 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:39.810+0000 7f1eafe95700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1ea8105470 0x7f1ea8105890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:08:40.809 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:39.810+0000 7f1eafe95700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ea8105e60 con 0x7f1ea8105470 2024-05-04T12:08:40.809 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:39.811+0000 7f1eadc31700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1ea8105470 0x7f1ea8105890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:08:40.809 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:39.811+0000 7f1eadc31700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1ea8105470 0x7f1ea8105890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:57230/0 (socket says 172.21.15.79:57230) 2024-05-04T12:08:40.809 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:39.811+0000 7f1eadc31700 1 -- 172.21.15.79:0/3885025274 learned_addr learned my addr 172.21.15.79:0/3885025274 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:08:40.809 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:39.811+0000 7f1eadc31700 1 -- 172.21.15.79:0/3885025274 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ea8106670 con 0x7f1ea8105470 2024-05-04T12:08:40.809 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:39.811+0000 7f1eadc31700 1 --2- 172.21.15.79:0/3885025274 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1ea8105470 0x7f1ea8105890 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f1ea4009e80 tx=0x7f1ea400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4795d7585eb8ee6 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:08:40.809 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:39.811+0000 7f1eacc2f700 1 -- 172.21.15.79:0/3885025274 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1ea400e070 con 0x7f1ea8105470 2024-05-04T12:08:40.809 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:39.811+0000 7f1eacc2f700 1 -- 172.21.15.79:0/3885025274 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f1ea400b8c0 con 0x7f1ea8105470 2024-05-04T12:08:40.810 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:39.811+0000 7f1eacc2f700 1 -- 172.21.15.79:0/3885025274 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1ea40044d0 con 0x7f1ea8105470 2024-05-04T12:08:40.810 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:39.812+0000 7f1eafe95700 1 -- 172.21.15.79:0/3885025274 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1ea8105470 msgr2=0x7f1ea8105890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:08:40.810 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:39.812+0000 7f1eafe95700 1 --2- 172.21.15.79:0/3885025274 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1ea8105470 0x7f1ea8105890 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f1ea4009e80 tx=0x7f1ea400b330 comp rx=0 tx=0).stop 2024-05-04T12:08:40.810 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:39.812+0000 7f1eafe95700 1 -- 172.21.15.79:0/3885025274 shutdown_connections 2024-05-04T12:08:40.810 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:39.812+0000 7f1eafe95700 1 --2- 172.21.15.79:0/3885025274 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1ea8105470 0x7f1ea8105890 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:08:40.810 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:39.812+0000 7f1eafe95700 1 -- 172.21.15.79:0/3885025274 >> 172.21.15.79:0/3885025274 conn(0x7f1ea81009f0 msgr2=0x7f1ea8102e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:08:40.810 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:39.812+0000 7f1eafe95700 1 -- 172.21.15.79:0/3885025274 shutdown_connections 2024-05-04T12:08:40.810 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:39.812+0000 7f1eafe95700 1 -- 172.21.15.79:0/3885025274 wait complete. 2024-05-04T12:08:40.810 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:39.813+0000 7f1eafe95700 1 Processor -- start 2024-05-04T12:08:40.810 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:39.813+0000 7f1eafe95700 1 -- start start 2024-05-04T12:08:40.810 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:39.813+0000 7f1eafe95700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1ea8105470 0x7f1ea81990a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:08:40.810 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:39.813+0000 7f1eafe95700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ea81995e0 con 0x7f1ea8105470 2024-05-04T12:08:40.810 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:39.813+0000 7f1eadc31700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1ea8105470 0x7f1ea81990a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:08:40.811 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:39.813+0000 7f1eadc31700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1ea8105470 0x7f1ea81990a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:57240/0 (socket says 172.21.15.79:57240) 2024-05-04T12:08:40.811 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:39.813+0000 7f1eadc31700 1 -- 172.21.15.79:0/1352487952 learned_addr learned my addr 172.21.15.79:0/1352487952 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:08:40.811 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:39.813+0000 7f1eadc31700 1 -- 172.21.15.79:0/1352487952 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ea4009b30 con 0x7f1ea8105470 2024-05-04T12:08:40.811 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:39.813+0000 7f1eadc31700 1 --2- 172.21.15.79:0/1352487952 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1ea8105470 0x7f1ea81990a0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f1ea4004940 tx=0x7f1ea4004a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:08:40.811 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:39.813+0000 7f1e9effd700 1 -- 172.21.15.79:0/1352487952 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1ea400e040 con 0x7f1ea8105470 2024-05-04T12:08:40.811 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:39.813+0000 7f1e9effd700 1 -- 172.21.15.79:0/1352487952 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f1ea400bc40 con 0x7f1ea8105470 2024-05-04T12:08:40.811 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:39.813+0000 7f1eafe95700 1 -- 172.21.15.79:0/1352487952 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ea81997e0 con 0x7f1ea8105470 2024-05-04T12:08:40.811 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:39.813+0000 7f1e9effd700 1 -- 172.21.15.79:0/1352487952 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1ea40126e0 con 0x7f1ea8105470 2024-05-04T12:08:40.811 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:39.814+0000 7f1eafe95700 1 -- 172.21.15.79:0/1352487952 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ea8199ce0 con 0x7f1ea8105470 2024-05-04T12:08:40.811 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:39.814+0000 7f1e9effd700 1 -- 172.21.15.79:0/1352487952 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7f1ea4019040 con 0x7f1ea8105470 2024-05-04T12:08:40.811 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:39.814+0000 7f1eafe95700 1 -- 172.21.15.79:0/1352487952 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ea80623c0 con 0x7f1ea8105470 2024-05-04T12:08:40.811 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:39.815+0000 7f1e9effd700 1 --2- 172.21.15.79:0/1352487952 >> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] conn(0x7f1e94038320 0x7f1e9403a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:08:40.811 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:39.815+0000 7f1e9effd700 1 -- 172.21.15.79:0/1352487952 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f1ea404bce0 con 0x7f1ea8105470 2024-05-04T12:08:40.811 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:39.815+0000 7f1ead430700 1 --2- 172.21.15.79:0/1352487952 >> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] conn(0x7f1e94038320 0x7f1e9403a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:08:40.812 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:39.815+0000 7f1ead430700 1 --2- 172.21.15.79:0/1352487952 >> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] conn(0x7f1e94038320 0x7f1e9403a7e0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f1e98006fd0 tx=0x7f1e98006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:08:40.812 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:39.819+0000 7f1e9effd700 1 -- 172.21.15.79:0/1352487952 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f1ea4011750 con 0x7f1ea8105470 2024-05-04T12:08:40.812 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:40.042+0000 7f1eafe95700 1 -- 172.21.15.79:0/1352487952 --> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}) v1 -- 0x7f1ea8101e50 con 0x7f1e94038320 2024-05-04T12:08:40.812 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:40.046+0000 7f1e9effd700 1 -- 172.21.15.79:0/1352487952 <== mgr.14120 v2:172.21.15.79:6800/1620705054 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7f1ea8101e50 con 0x7f1e94038320 2024-05-04T12:08:40.812 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:40.048+0000 7f1eafe95700 1 -- 172.21.15.79:0/1352487952 >> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] conn(0x7f1e94038320 msgr2=0x7f1e9403a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:08:40.812 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:40.048+0000 7f1eafe95700 1 --2- 172.21.15.79:0/1352487952 >> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] conn(0x7f1e94038320 0x7f1e9403a7e0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f1e98006fd0 tx=0x7f1e98006e40 comp rx=0 tx=0).stop 2024-05-04T12:08:40.812 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:40.048+0000 7f1eafe95700 1 -- 172.21.15.79:0/1352487952 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1ea8105470 msgr2=0x7f1ea81990a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:08:40.812 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:40.048+0000 7f1eafe95700 1 --2- 172.21.15.79:0/1352487952 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1ea8105470 0x7f1ea81990a0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f1ea4004940 tx=0x7f1ea4004a20 comp rx=0 tx=0).stop 2024-05-04T12:08:40.812 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:40.049+0000 7f1eafe95700 1 -- 172.21.15.79:0/1352487952 shutdown_connections 2024-05-04T12:08:40.812 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:40.049+0000 7f1eafe95700 1 --2- 172.21.15.79:0/1352487952 >> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] conn(0x7f1e94038320 0x7f1e9403a7e0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:08:40.812 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:40.049+0000 7f1eafe95700 1 --2- 172.21.15.79:0/1352487952 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1ea8105470 0x7f1ea81990a0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:08:40.812 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:40.049+0000 7f1eafe95700 1 -- 172.21.15.79:0/1352487952 >> 172.21.15.79:0/1352487952 conn(0x7f1ea81009f0 msgr2=0x7f1ea8101740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:08:40.812 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:40.049+0000 7f1eafe95700 1 -- 172.21.15.79:0/1352487952 shutdown_connections 2024-05-04T12:08:40.812 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:40.049+0000 7f1eafe95700 1 -- 172.21.15.79:0/1352487952 wait complete. 2024-05-04T12:08:40.812 INFO:teuthology.orchestra.run.smithi079.stdout:Deploying ceph-exporter service with default placement... 2024-05-04T12:08:41.432 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:08:41 smithi079 ceph-mon[27222]: from='client.14146 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2024-05-04T12:08:41.432 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:08:41 smithi079 ceph-mon[27222]: Saving service crash spec with placement * 2024-05-04T12:08:42.083 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout Scheduled ceph-exporter update... 2024-05-04T12:08:42.083 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:41.220+0000 7f0743623700 1 Processor -- start 2024-05-04T12:08:42.083 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:41.220+0000 7f0743623700 1 -- start start 2024-05-04T12:08:42.083 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:41.220+0000 7f0743623700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f073c107060 0x7f073c109450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:08:42.083 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:41.220+0000 7f0743623700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f073c074600 con 0x7f073c107060 2024-05-04T12:08:42.084 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:41.221+0000 7f07413bf700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f073c107060 0x7f073c109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:08:42.084 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:41.221+0000 7f07413bf700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f073c107060 0x7f073c109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:57252/0 (socket says 172.21.15.79:57252) 2024-05-04T12:08:42.084 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:41.221+0000 7f07413bf700 1 -- 172.21.15.79:0/4294700245 learned_addr learned my addr 172.21.15.79:0/4294700245 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:08:42.084 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:41.221+0000 7f07413bf700 1 -- 172.21.15.79:0/4294700245 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f073c109990 con 0x7f073c107060 2024-05-04T12:08:42.084 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:41.221+0000 7f07413bf700 1 --2- 172.21.15.79:0/4294700245 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f073c107060 0x7f073c109450 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f07380098a0 tx=0x7f0738009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4362c60b060eb06 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:08:42.084 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:41.221+0000 7f0733fff700 1 -- 172.21.15.79:0/4294700245 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0738004180 con 0x7f073c107060 2024-05-04T12:08:42.084 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:41.221+0000 7f0733fff700 1 -- 172.21.15.79:0/4294700245 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f07380042e0 con 0x7f073c107060 2024-05-04T12:08:42.084 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:41.221+0000 7f0733fff700 1 -- 172.21.15.79:0/4294700245 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0738011af0 con 0x7f073c107060 2024-05-04T12:08:42.085 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:41.222+0000 7f0743623700 1 -- 172.21.15.79:0/4294700245 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f073c107060 msgr2=0x7f073c109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:08:42.085 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:41.222+0000 7f0743623700 1 --2- 172.21.15.79:0/4294700245 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f073c107060 0x7f073c109450 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f07380098a0 tx=0x7f0738009bb0 comp rx=0 tx=0).stop 2024-05-04T12:08:42.085 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:41.222+0000 7f0743623700 1 -- 172.21.15.79:0/4294700245 shutdown_connections 2024-05-04T12:08:42.085 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:41.222+0000 7f0743623700 1 --2- 172.21.15.79:0/4294700245 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f073c107060 0x7f073c109450 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:08:42.085 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:41.222+0000 7f0743623700 1 -- 172.21.15.79:0/4294700245 >> 172.21.15.79:0/4294700245 conn(0x7f073c1009f0 msgr2=0x7f073c102e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:08:42.086 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:41.222+0000 7f0743623700 1 -- 172.21.15.79:0/4294700245 shutdown_connections 2024-05-04T12:08:42.086 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:41.222+0000 7f0743623700 1 -- 172.21.15.79:0/4294700245 wait complete. 2024-05-04T12:08:42.086 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:41.222+0000 7f0743623700 1 Processor -- start 2024-05-04T12:08:42.086 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:41.222+0000 7f0743623700 1 -- start start 2024-05-04T12:08:42.086 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:41.223+0000 7f0743623700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f073c107060 0x7f073c19c390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:08:42.086 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:41.223+0000 7f0743623700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f073c19c8d0 con 0x7f073c107060 2024-05-04T12:08:42.087 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:41.223+0000 7f07413bf700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f073c107060 0x7f073c19c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:08:42.087 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:41.223+0000 7f07413bf700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f073c107060 0x7f073c19c390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:57258/0 (socket says 172.21.15.79:57258) 2024-05-04T12:08:42.087 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:41.223+0000 7f07413bf700 1 -- 172.21.15.79:0/2335382408 learned_addr learned my addr 172.21.15.79:0/2335382408 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:08:42.087 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:41.223+0000 7f07413bf700 1 -- 172.21.15.79:0/2335382408 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0738009550 con 0x7f073c107060 2024-05-04T12:08:42.087 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:41.223+0000 7f07413bf700 1 --2- 172.21.15.79:0/2335382408 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f073c107060 0x7f073c19c390 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f0738011ad0 tx=0x7f0738011bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:08:42.087 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:41.223+0000 7f07327fc700 1 -- 172.21.15.79:0/2335382408 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0738003ab0 con 0x7f073c107060 2024-05-04T12:08:42.088 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:41.223+0000 7f07327fc700 1 -- 172.21.15.79:0/2335382408 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f0738003c10 con 0x7f073c107060 2024-05-04T12:08:42.088 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:41.223+0000 7f07327fc700 1 -- 172.21.15.79:0/2335382408 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f07380198e0 con 0x7f073c107060 2024-05-04T12:08:42.088 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:41.223+0000 7f0743623700 1 -- 172.21.15.79:0/2335382408 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f073c19cad0 con 0x7f073c107060 2024-05-04T12:08:42.088 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:41.223+0000 7f0743623700 1 -- 172.21.15.79:0/2335382408 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f073c19cf70 con 0x7f073c107060 2024-05-04T12:08:42.088 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:41.224+0000 7f07327fc700 1 -- 172.21.15.79:0/2335382408 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7f0738019ba0 con 0x7f073c107060 2024-05-04T12:08:42.088 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:41.224+0000 7f0743623700 1 -- 172.21.15.79:0/2335382408 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f073c10a6d0 con 0x7f073c107060 2024-05-04T12:08:42.088 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:41.224+0000 7f07327fc700 1 --2- 172.21.15.79:0/2335382408 >> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] conn(0x7f0728038320 0x7f072803a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:08:42.089 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:41.225+0000 7f07327fc700 1 -- 172.21.15.79:0/2335382408 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f073804c170 con 0x7f073c107060 2024-05-04T12:08:42.089 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:41.225+0000 7f0740bbe700 1 --2- 172.21.15.79:0/2335382408 >> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] conn(0x7f0728038320 0x7f072803a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:08:42.089 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:41.225+0000 7f0740bbe700 1 --2- 172.21.15.79:0/2335382408 >> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] conn(0x7f0728038320 0x7f072803a7e0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f072c006fd0 tx=0x7f072c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:08:42.089 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:41.229+0000 7f07327fc700 1 -- 172.21.15.79:0/2335382408 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f073801a430 con 0x7f073c107060 2024-05-04T12:08:42.089 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:41.433+0000 7f0743623700 1 -- 172.21.15.79:0/2335382408 --> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "ceph-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7f073c101e10 con 0x7f0728038320 2024-05-04T12:08:42.089 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:41.438+0000 7f07327fc700 1 -- 172.21.15.79:0/2335382408 <== mgr.14120 v2:172.21.15.79:6800/1620705054 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7f073c101e10 con 0x7f0728038320 2024-05-04T12:08:42.090 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:41.440+0000 7f0743623700 1 -- 172.21.15.79:0/2335382408 >> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] conn(0x7f0728038320 msgr2=0x7f072803a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:08:42.090 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:41.440+0000 7f0743623700 1 --2- 172.21.15.79:0/2335382408 >> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] conn(0x7f0728038320 0x7f072803a7e0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f072c006fd0 tx=0x7f072c006e40 comp rx=0 tx=0).stop 2024-05-04T12:08:42.090 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:41.440+0000 7f0743623700 1 -- 172.21.15.79:0/2335382408 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f073c107060 msgr2=0x7f073c19c390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:08:42.090 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:41.440+0000 7f0743623700 1 --2- 172.21.15.79:0/2335382408 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f073c107060 0x7f073c19c390 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f0738011ad0 tx=0x7f0738011bb0 comp rx=0 tx=0).stop 2024-05-04T12:08:42.090 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:41.440+0000 7f0743623700 1 -- 172.21.15.79:0/2335382408 shutdown_connections 2024-05-04T12:08:42.091 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:41.440+0000 7f0743623700 1 --2- 172.21.15.79:0/2335382408 >> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] conn(0x7f0728038320 0x7f072803a7e0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:08:42.091 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:41.440+0000 7f0743623700 1 --2- 172.21.15.79:0/2335382408 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f073c107060 0x7f073c19c390 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:08:42.091 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:41.440+0000 7f0743623700 1 -- 172.21.15.79:0/2335382408 >> 172.21.15.79:0/2335382408 conn(0x7f073c1009f0 msgr2=0x7f073c101700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:08:42.091 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:41.440+0000 7f0743623700 1 -- 172.21.15.79:0/2335382408 shutdown_connections 2024-05-04T12:08:42.091 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:41.440+0000 7f0743623700 1 -- 172.21.15.79:0/2335382408 wait complete. 2024-05-04T12:08:42.091 INFO:teuthology.orchestra.run.smithi079.stdout:Deploying prometheus service with default placement... 2024-05-04T12:08:42.478 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:08:42 smithi079 ceph-mon[27222]: from='client.14148 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "ceph-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-05-04T12:08:42.478 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:08:42 smithi079 ceph-mon[27222]: Saving service ceph-exporter spec with placement * 2024-05-04T12:08:42.478 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:08:42 smithi079 ceph-mon[27222]: from='mgr.14120 172.21.15.79:0/2394226123' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:08:43.336 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout Scheduled prometheus update... 2024-05-04T12:08:43.337 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:42.508+0000 7f2392c83700 1 Processor -- start 2024-05-04T12:08:43.337 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:42.508+0000 7f2392c83700 1 -- start start 2024-05-04T12:08:43.337 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:42.508+0000 7f2392c83700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f238c07bea0 0x7f238c07a300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:08:43.337 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:42.508+0000 7f2392c83700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f238c07a8d0 con 0x7f238c07bea0 2024-05-04T12:08:43.337 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:42.508+0000 7f2390a1f700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f238c07bea0 0x7f238c07a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:08:43.337 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:42.508+0000 7f2390a1f700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f238c07bea0 0x7f238c07a300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:57266/0 (socket says 172.21.15.79:57266) 2024-05-04T12:08:43.337 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:42.508+0000 7f2390a1f700 1 -- 172.21.15.79:0/178778934 learned_addr learned my addr 172.21.15.79:0/178778934 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:08:43.337 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:42.508+0000 7f2390a1f700 1 -- 172.21.15.79:0/178778934 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f238c07aa10 con 0x7f238c07bea0 2024-05-04T12:08:43.338 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:42.509+0000 7f2390a1f700 1 --2- 172.21.15.79:0/178778934 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f238c07bea0 0x7f238c07a300 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f23800098a0 tx=0x7f2380009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e940fb942d89362d server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:08:43.338 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:42.509+0000 7f238b7fe700 1 -- 172.21.15.79:0/178778934 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2380004180 con 0x7f238c07bea0 2024-05-04T12:08:43.338 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:42.509+0000 7f238b7fe700 1 -- 172.21.15.79:0/178778934 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f23800042e0 con 0x7f238c07bea0 2024-05-04T12:08:43.338 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:42.509+0000 7f238b7fe700 1 -- 172.21.15.79:0/178778934 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2380011af0 con 0x7f238c07bea0 2024-05-04T12:08:43.338 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:42.509+0000 7f2392c83700 1 -- 172.21.15.79:0/178778934 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f238c07bea0 msgr2=0x7f238c07a300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:08:43.338 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:42.509+0000 7f2392c83700 1 --2- 172.21.15.79:0/178778934 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f238c07bea0 0x7f238c07a300 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f23800098a0 tx=0x7f2380009bb0 comp rx=0 tx=0).stop 2024-05-04T12:08:43.338 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:42.509+0000 7f2392c83700 1 -- 172.21.15.79:0/178778934 shutdown_connections 2024-05-04T12:08:43.339 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:42.509+0000 7f2392c83700 1 --2- 172.21.15.79:0/178778934 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f238c07bea0 0x7f238c07a300 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:08:43.339 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:42.509+0000 7f2392c83700 1 -- 172.21.15.79:0/178778934 >> 172.21.15.79:0/178778934 conn(0x7f238c101b40 msgr2=0x7f238c103f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:08:43.339 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:42.510+0000 7f2392c83700 1 -- 172.21.15.79:0/178778934 shutdown_connections 2024-05-04T12:08:43.339 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:42.510+0000 7f2392c83700 1 -- 172.21.15.79:0/178778934 wait complete. 2024-05-04T12:08:43.339 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:42.510+0000 7f2392c83700 1 Processor -- start 2024-05-04T12:08:43.339 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:42.510+0000 7f2392c83700 1 -- start start 2024-05-04T12:08:43.339 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:42.510+0000 7f2392c83700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f238c07bea0 0x7f238c19c460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:08:43.339 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:42.510+0000 7f2392c83700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f238c19c9a0 con 0x7f238c07bea0 2024-05-04T12:08:43.339 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:42.510+0000 7f2390a1f700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f238c07bea0 0x7f238c19c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:08:43.340 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:42.510+0000 7f2390a1f700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f238c07bea0 0x7f238c19c460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:57274/0 (socket says 172.21.15.79:57274) 2024-05-04T12:08:43.340 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:42.510+0000 7f2390a1f700 1 -- 172.21.15.79:0/3254605835 learned_addr learned my addr 172.21.15.79:0/3254605835 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:08:43.340 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:42.511+0000 7f2390a1f700 1 -- 172.21.15.79:0/3254605835 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2380009550 con 0x7f238c07bea0 2024-05-04T12:08:43.340 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:42.511+0000 7f2390a1f700 1 --2- 172.21.15.79:0/3254605835 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f238c07bea0 0x7f238c19c460 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f2380011670 tx=0x7f2380011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:08:43.340 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:42.511+0000 7f2389ffb700 1 -- 172.21.15.79:0/3254605835 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f23800039c0 con 0x7f238c07bea0 2024-05-04T12:08:43.340 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:42.511+0000 7f2389ffb700 1 -- 172.21.15.79:0/3254605835 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f2380003b20 con 0x7f238c07bea0 2024-05-04T12:08:43.340 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:42.511+0000 7f2392c83700 1 -- 172.21.15.79:0/3254605835 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f238c19cba0 con 0x7f238c07bea0 2024-05-04T12:08:43.340 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:42.511+0000 7f2389ffb700 1 -- 172.21.15.79:0/3254605835 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f23800247f0 con 0x7f238c07bea0 2024-05-04T12:08:43.340 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:42.511+0000 7f2392c83700 1 -- 172.21.15.79:0/3254605835 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f238c19d040 con 0x7f238c07bea0 2024-05-04T12:08:43.340 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:42.512+0000 7f2389ffb700 1 -- 172.21.15.79:0/3254605835 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7f238001a430 con 0x7f238c07bea0 2024-05-04T12:08:43.341 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:42.512+0000 7f2392c83700 1 -- 172.21.15.79:0/3254605835 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f238c10c900 con 0x7f238c07bea0 2024-05-04T12:08:43.341 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:42.512+0000 7f2389ffb700 1 --2- 172.21.15.79:0/3254605835 >> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] conn(0x7f237c038330 0x7f237c03a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:08:43.341 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:42.512+0000 7f2389ffb700 1 -- 172.21.15.79:0/3254605835 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f238004c470 con 0x7f238c07bea0 2024-05-04T12:08:43.341 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:42.512+0000 7f238bfff700 1 --2- 172.21.15.79:0/3254605835 >> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] conn(0x7f237c038330 0x7f237c03a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:08:43.341 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:42.513+0000 7f238bfff700 1 --2- 172.21.15.79:0/3254605835 >> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] conn(0x7f237c038330 0x7f237c03a7f0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f2378006fd0 tx=0x7f2378006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:08:43.341 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:42.517+0000 7f2389ffb700 1 -- 172.21.15.79:0/3254605835 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f238001a6e0 con 0x7f238c07bea0 2024-05-04T12:08:43.341 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:42.727+0000 7f2392c83700 1 -- 172.21.15.79:0/3254605835 --> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7f238c103df0 con 0x7f237c038330 2024-05-04T12:08:43.341 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:42.733+0000 7f2389ffb700 1 -- 172.21.15.79:0/3254605835 <== mgr.14120 v2:172.21.15.79:6800/1620705054 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+31 (secure 0 0 0) 0x7f238c103df0 con 0x7f237c038330 2024-05-04T12:08:43.341 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:42.735+0000 7f2392c83700 1 -- 172.21.15.79:0/3254605835 >> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] conn(0x7f237c038330 msgr2=0x7f237c03a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:08:43.341 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:42.735+0000 7f2392c83700 1 --2- 172.21.15.79:0/3254605835 >> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] conn(0x7f237c038330 0x7f237c03a7f0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f2378006fd0 tx=0x7f2378006e40 comp rx=0 tx=0).stop 2024-05-04T12:08:43.341 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:42.735+0000 7f2392c83700 1 -- 172.21.15.79:0/3254605835 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f238c07bea0 msgr2=0x7f238c19c460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:08:43.341 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:42.735+0000 7f2392c83700 1 --2- 172.21.15.79:0/3254605835 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f238c07bea0 0x7f238c19c460 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f2380011670 tx=0x7f2380011aa0 comp rx=0 tx=0).stop 2024-05-04T12:08:43.341 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:42.735+0000 7f2392c83700 1 -- 172.21.15.79:0/3254605835 shutdown_connections 2024-05-04T12:08:43.341 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:42.735+0000 7f2392c83700 1 --2- 172.21.15.79:0/3254605835 >> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] conn(0x7f237c038330 0x7f237c03a7f0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:08:43.342 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:42.735+0000 7f2392c83700 1 --2- 172.21.15.79:0/3254605835 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f238c07bea0 0x7f238c19c460 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:08:43.342 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:42.735+0000 7f2392c83700 1 -- 172.21.15.79:0/3254605835 >> 172.21.15.79:0/3254605835 conn(0x7f238c101b40 msgr2=0x7f238c103190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:08:43.342 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:42.735+0000 7f2392c83700 1 -- 172.21.15.79:0/3254605835 shutdown_connections 2024-05-04T12:08:43.342 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:42.735+0000 7f2392c83700 1 -- 172.21.15.79:0/3254605835 wait complete. 2024-05-04T12:08:43.342 INFO:teuthology.orchestra.run.smithi079.stdout:Deploying grafana service with default placement... 2024-05-04T12:08:43.945 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:08:43 smithi079 ceph-mon[27222]: from='mgr.14120 172.21.15.79:0/2394226123' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:08:43.945 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:08:43 smithi079 ceph-mon[27222]: from='client.14150 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}]: dispatch 2024-05-04T12:08:43.945 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:08:43 smithi079 ceph-mon[27222]: Saving service prometheus spec with placement count:1 2024-05-04T12:08:43.945 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:08:43 smithi079 ceph-mon[27222]: from='mgr.14120 172.21.15.79:0/2394226123' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:08:44.531 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout Scheduled grafana update... 2024-05-04T12:08:44.531 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:43.794+0000 7fc855faa700 1 Processor -- start 2024-05-04T12:08:44.531 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:43.794+0000 7fc855faa700 1 -- start start 2024-05-04T12:08:44.531 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:43.794+0000 7fc855faa700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc85007bea0 0x7fc85007a300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:08:44.531 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:43.794+0000 7fc855faa700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc85007a8d0 con 0x7fc85007bea0 2024-05-04T12:08:44.531 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:43.795+0000 7fc84f7fe700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc85007bea0 0x7fc85007a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:08:44.531 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:43.795+0000 7fc84f7fe700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc85007bea0 0x7fc85007a300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:34284/0 (socket says 172.21.15.79:34284) 2024-05-04T12:08:44.531 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:43.795+0000 7fc84f7fe700 1 -- 172.21.15.79:0/3808734638 learned_addr learned my addr 172.21.15.79:0/3808734638 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:08:44.531 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:43.795+0000 7fc84f7fe700 1 -- 172.21.15.79:0/3808734638 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc85007aa10 con 0x7fc85007bea0 2024-05-04T12:08:44.531 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:43.795+0000 7fc84f7fe700 1 --2- 172.21.15.79:0/3808734638 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc85007bea0 0x7fc85007a300 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fc8440098a0 tx=0x7fc844009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62fbfca50f212479 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:08:44.531 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:43.795+0000 7fc84e7fc700 1 -- 172.21.15.79:0/3808734638 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc844004180 con 0x7fc85007bea0 2024-05-04T12:08:44.532 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:43.795+0000 7fc84e7fc700 1 -- 172.21.15.79:0/3808734638 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fc8440042e0 con 0x7fc85007bea0 2024-05-04T12:08:44.532 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:43.795+0000 7fc84e7fc700 1 -- 172.21.15.79:0/3808734638 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc844011af0 con 0x7fc85007bea0 2024-05-04T12:08:44.532 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:43.796+0000 7fc855faa700 1 -- 172.21.15.79:0/3808734638 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc85007bea0 msgr2=0x7fc85007a300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:08:44.532 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:43.796+0000 7fc855faa700 1 --2- 172.21.15.79:0/3808734638 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc85007bea0 0x7fc85007a300 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fc8440098a0 tx=0x7fc844009bb0 comp rx=0 tx=0).stop 2024-05-04T12:08:44.532 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:43.796+0000 7fc855faa700 1 -- 172.21.15.79:0/3808734638 shutdown_connections 2024-05-04T12:08:44.532 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:43.796+0000 7fc855faa700 1 --2- 172.21.15.79:0/3808734638 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc85007bea0 0x7fc85007a300 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:08:44.532 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:43.796+0000 7fc855faa700 1 -- 172.21.15.79:0/3808734638 >> 172.21.15.79:0/3808734638 conn(0x7fc850101b40 msgr2=0x7fc850103f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:08:44.532 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:43.796+0000 7fc855faa700 1 -- 172.21.15.79:0/3808734638 shutdown_connections 2024-05-04T12:08:44.532 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:43.796+0000 7fc855faa700 1 -- 172.21.15.79:0/3808734638 wait complete. 2024-05-04T12:08:44.532 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:43.797+0000 7fc855faa700 1 Processor -- start 2024-05-04T12:08:44.532 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:43.797+0000 7fc855faa700 1 -- start start 2024-05-04T12:08:44.533 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:43.797+0000 7fc855faa700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc85007bea0 0x7fc85019c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:08:44.533 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:43.797+0000 7fc855faa700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc85019c910 con 0x7fc85007bea0 2024-05-04T12:08:44.533 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:43.797+0000 7fc84f7fe700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc85007bea0 0x7fc85019c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:08:44.533 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:43.797+0000 7fc84f7fe700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc85007bea0 0x7fc85019c3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:34294/0 (socket says 172.21.15.79:34294) 2024-05-04T12:08:44.533 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:43.797+0000 7fc84f7fe700 1 -- 172.21.15.79:0/2130796515 learned_addr learned my addr 172.21.15.79:0/2130796515 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:08:44.533 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:43.798+0000 7fc84f7fe700 1 -- 172.21.15.79:0/2130796515 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc844009550 con 0x7fc85007bea0 2024-05-04T12:08:44.533 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:43.798+0000 7fc84f7fe700 1 --2- 172.21.15.79:0/2130796515 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc85007bea0 0x7fc85019c3d0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fc844000c00 tx=0x7fc844011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:08:44.533 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:43.798+0000 7fc84cff9700 1 -- 172.21.15.79:0/2130796515 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc844003a70 con 0x7fc85007bea0 2024-05-04T12:08:44.533 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:43.798+0000 7fc84cff9700 1 -- 172.21.15.79:0/2130796515 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fc844003bd0 con 0x7fc85007bea0 2024-05-04T12:08:44.533 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:43.798+0000 7fc855faa700 1 -- 172.21.15.79:0/2130796515 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc85019cb10 con 0x7fc85007bea0 2024-05-04T12:08:44.534 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:43.798+0000 7fc84cff9700 1 -- 172.21.15.79:0/2130796515 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc8440238c0 con 0x7fc85007bea0 2024-05-04T12:08:44.534 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:43.798+0000 7fc855faa700 1 -- 172.21.15.79:0/2130796515 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc85019cfb0 con 0x7fc85007bea0 2024-05-04T12:08:44.534 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:43.799+0000 7fc84cff9700 1 -- 172.21.15.79:0/2130796515 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7fc844023b30 con 0x7fc85007bea0 2024-05-04T12:08:44.534 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:43.799+0000 7fc84cff9700 1 --2- 172.21.15.79:0/2130796515 >> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] conn(0x7fc838038370 0x7fc83803a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:08:44.534 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:43.799+0000 7fc855faa700 1 -- 172.21.15.79:0/2130796515 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc85010c900 con 0x7fc85007bea0 2024-05-04T12:08:44.534 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:43.799+0000 7fc84cff9700 1 -- 172.21.15.79:0/2130796515 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fc84404c1a0 con 0x7fc85007bea0 2024-05-04T12:08:44.534 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:43.799+0000 7fc84effd700 1 --2- 172.21.15.79:0/2130796515 >> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] conn(0x7fc838038370 0x7fc83803a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:08:44.534 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:43.800+0000 7fc84effd700 1 --2- 172.21.15.79:0/2130796515 >> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] conn(0x7fc838038370 0x7fc83803a830 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fc840006fd0 tx=0x7fc840006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:08:44.534 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:43.804+0000 7fc84cff9700 1 -- 172.21.15.79:0/2130796515 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fc844010c20 con 0x7fc85007bea0 2024-05-04T12:08:44.535 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:44.013+0000 7fc855faa700 1 -- 172.21.15.79:0/2130796515 --> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}) v1 -- 0x7fc850102fc0 con 0x7fc838038370 2024-05-04T12:08:44.535 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:44.019+0000 7fc84cff9700 1 -- 172.21.15.79:0/2130796515 <== mgr.14120 v2:172.21.15.79:6800/1620705054 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7fc850102fc0 con 0x7fc838038370 2024-05-04T12:08:44.535 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:44.021+0000 7fc855faa700 1 -- 172.21.15.79:0/2130796515 >> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] conn(0x7fc838038370 msgr2=0x7fc83803a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:08:44.535 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:44.021+0000 7fc855faa700 1 --2- 172.21.15.79:0/2130796515 >> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] conn(0x7fc838038370 0x7fc83803a830 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fc840006fd0 tx=0x7fc840006e40 comp rx=0 tx=0).stop 2024-05-04T12:08:44.535 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:44.021+0000 7fc855faa700 1 -- 172.21.15.79:0/2130796515 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc85007bea0 msgr2=0x7fc85019c3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:08:44.535 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:44.021+0000 7fc855faa700 1 --2- 172.21.15.79:0/2130796515 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc85007bea0 0x7fc85019c3d0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fc844000c00 tx=0x7fc844011b50 comp rx=0 tx=0).stop 2024-05-04T12:08:44.535 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:44.021+0000 7fc855faa700 1 -- 172.21.15.79:0/2130796515 shutdown_connections 2024-05-04T12:08:44.535 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:44.021+0000 7fc855faa700 1 --2- 172.21.15.79:0/2130796515 >> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] conn(0x7fc838038370 0x7fc83803a830 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:08:44.535 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:44.021+0000 7fc855faa700 1 --2- 172.21.15.79:0/2130796515 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc85007bea0 0x7fc85019c3d0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:08:44.535 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:44.021+0000 7fc855faa700 1 -- 172.21.15.79:0/2130796515 >> 172.21.15.79:0/2130796515 conn(0x7fc850101b40 msgr2=0x7fc8501028b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:08:44.535 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:44.021+0000 7fc855faa700 1 -- 172.21.15.79:0/2130796515 shutdown_connections 2024-05-04T12:08:44.535 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:44.021+0000 7fc855faa700 1 -- 172.21.15.79:0/2130796515 wait complete. 2024-05-04T12:08:44.535 INFO:teuthology.orchestra.run.smithi079.stdout:Deploying node-exporter service with default placement... 2024-05-04T12:08:44.933 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:08:44 smithi079 ceph-mon[27222]: from='mgr.14120 172.21.15.79:0/2394226123' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:08:44.933 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:08:44 smithi079 ceph-mon[27222]: from='client.14152 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2024-05-04T12:08:44.933 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:08:44 smithi079 ceph-mon[27222]: Saving service grafana spec with placement count:1 2024-05-04T12:08:44.933 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:08:44 smithi079 ceph-mon[27222]: from='mgr.14120 172.21.15.79:0/2394226123' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:08:46.259 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:08:46 smithi079 ceph-mon[27222]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-05-04T12:08:46.259 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:08:46 smithi079 ceph-mon[27222]: Saving service node-exporter spec with placement * 2024-05-04T12:08:46.259 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:08:46 smithi079 ceph-mon[27222]: from='mgr.14120 172.21.15.79:0/2394226123' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:08:46.568 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout Scheduled node-exporter update... 2024-05-04T12:08:46.569 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:44.948+0000 7f370b995700 1 Processor -- start 2024-05-04T12:08:46.569 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:44.949+0000 7f370b995700 1 -- start start 2024-05-04T12:08:46.569 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:44.949+0000 7f370b995700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3704105460 0x7f3704105880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:08:46.569 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:44.949+0000 7f370b995700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3704105e50 con 0x7f3704105460 2024-05-04T12:08:46.569 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:44.949+0000 7f3709731700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3704105460 0x7f3704105880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:08:46.569 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:44.949+0000 7f3709731700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3704105460 0x7f3704105880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:34306/0 (socket says 172.21.15.79:34306) 2024-05-04T12:08:46.569 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:44.949+0000 7f3709731700 1 -- 172.21.15.79:0/885129212 learned_addr learned my addr 172.21.15.79:0/885129212 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:08:46.569 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:44.949+0000 7f3709731700 1 -- 172.21.15.79:0/885129212 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3704106660 con 0x7f3704105460 2024-05-04T12:08:46.569 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:44.949+0000 7f3709731700 1 --2- 172.21.15.79:0/885129212 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3704105460 0x7f3704105880 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f36f40098a0 tx=0x7f36f4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d22431bd43f0938 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:08:46.569 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:44.950+0000 7f36fbfff700 1 -- 172.21.15.79:0/885129212 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f36f4004180 con 0x7f3704105460 2024-05-04T12:08:46.569 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:44.950+0000 7f36fbfff700 1 -- 172.21.15.79:0/885129212 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f36f40042e0 con 0x7f3704105460 2024-05-04T12:08:46.569 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:44.950+0000 7f36fbfff700 1 -- 172.21.15.79:0/885129212 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f36f4011af0 con 0x7f3704105460 2024-05-04T12:08:46.570 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:44.950+0000 7f370b995700 1 -- 172.21.15.79:0/885129212 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3704105460 msgr2=0x7f3704105880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:08:46.570 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:44.950+0000 7f370b995700 1 --2- 172.21.15.79:0/885129212 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3704105460 0x7f3704105880 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f36f40098a0 tx=0x7f36f4009bb0 comp rx=0 tx=0).stop 2024-05-04T12:08:46.570 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:44.950+0000 7f370b995700 1 -- 172.21.15.79:0/885129212 shutdown_connections 2024-05-04T12:08:46.570 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:44.950+0000 7f370b995700 1 --2- 172.21.15.79:0/885129212 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3704105460 0x7f3704105880 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:08:46.570 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:44.950+0000 7f370b995700 1 -- 172.21.15.79:0/885129212 >> 172.21.15.79:0/885129212 conn(0x7f3704100a20 msgr2=0x7f3704102e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:08:46.570 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:44.950+0000 7f370b995700 1 -- 172.21.15.79:0/885129212 shutdown_connections 2024-05-04T12:08:46.570 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:44.950+0000 7f370b995700 1 -- 172.21.15.79:0/885129212 wait complete. 2024-05-04T12:08:46.570 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:44.951+0000 7f370b995700 1 Processor -- start 2024-05-04T12:08:46.570 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:44.951+0000 7f370b995700 1 -- start start 2024-05-04T12:08:46.570 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:44.951+0000 7f370b995700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3704105460 0x7f3704079b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:08:46.570 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:44.951+0000 7f370b995700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f370419cf90 con 0x7f3704105460 2024-05-04T12:08:46.570 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:44.951+0000 7f3709731700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3704105460 0x7f3704079b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:08:46.570 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:44.951+0000 7f3709731700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3704105460 0x7f3704079b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:34316/0 (socket says 172.21.15.79:34316) 2024-05-04T12:08:46.570 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:44.951+0000 7f3709731700 1 -- 172.21.15.79:0/179134587 learned_addr learned my addr 172.21.15.79:0/179134587 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:08:46.571 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:44.951+0000 7f3709731700 1 -- 172.21.15.79:0/179134587 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36f4009550 con 0x7f3704105460 2024-05-04T12:08:46.571 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:44.952+0000 7f3709731700 1 --2- 172.21.15.79:0/179134587 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3704105460 0x7f3704079b10 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f36f4011670 tx=0x7f36f4011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:08:46.571 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:44.952+0000 7f36fa7fc700 1 -- 172.21.15.79:0/179134587 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f36f40039f0 con 0x7f3704105460 2024-05-04T12:08:46.571 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:44.952+0000 7f36fa7fc700 1 -- 172.21.15.79:0/179134587 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f36f4003b50 con 0x7f3704105460 2024-05-04T12:08:46.571 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:44.952+0000 7f370b995700 1 -- 172.21.15.79:0/179134587 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f370419d0d0 con 0x7f3704105460 2024-05-04T12:08:46.571 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:44.952+0000 7f36fa7fc700 1 -- 172.21.15.79:0/179134587 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f36f40247c0 con 0x7f3704105460 2024-05-04T12:08:46.571 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:44.952+0000 7f370b995700 1 -- 172.21.15.79:0/179134587 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f370419d570 con 0x7f3704105460 2024-05-04T12:08:46.571 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:44.953+0000 7f36fa7fc700 1 -- 172.21.15.79:0/179134587 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7f36f401ac00 con 0x7f3704105460 2024-05-04T12:08:46.571 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:44.953+0000 7f370b995700 1 -- 172.21.15.79:0/179134587 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37040623c0 con 0x7f3704105460 2024-05-04T12:08:46.571 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:44.953+0000 7f36fa7fc700 1 --2- 172.21.15.79:0/179134587 >> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] conn(0x7f36f0038370 0x7f36f003a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:08:46.571 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:44.953+0000 7f36fa7fc700 1 -- 172.21.15.79:0/179134587 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f36f404c390 con 0x7f3704105460 2024-05-04T12:08:46.571 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:44.953+0000 7f3708f30700 1 --2- 172.21.15.79:0/179134587 >> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] conn(0x7f36f0038370 0x7f36f003a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:08:46.571 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:44.953+0000 7f3708f30700 1 --2- 172.21.15.79:0/179134587 >> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] conn(0x7f36f0038370 0x7f36f003a830 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f3700006fd0 tx=0x7f3700006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:08:46.571 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:44.958+0000 7f36fa7fc700 1 -- 172.21.15.79:0/179134587 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f36f401a430 con 0x7f3704105460 2024-05-04T12:08:46.572 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:45.175+0000 7f370b995700 1 -- 172.21.15.79:0/179134587 --> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7f3704109860 con 0x7f36f0038370 2024-05-04T12:08:46.572 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:45.180+0000 7f36fa7fc700 1 -- 172.21.15.79:0/179134587 <== mgr.14120 v2:172.21.15.79:6800/1620705054 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7f3704109860 con 0x7f36f0038370 2024-05-04T12:08:46.572 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:45.182+0000 7f370b995700 1 -- 172.21.15.79:0/179134587 >> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] conn(0x7f36f0038370 msgr2=0x7f36f003a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:08:46.572 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:45.182+0000 7f370b995700 1 --2- 172.21.15.79:0/179134587 >> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] conn(0x7f36f0038370 0x7f36f003a830 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f3700006fd0 tx=0x7f3700006e40 comp rx=0 tx=0).stop 2024-05-04T12:08:46.572 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:45.182+0000 7f370b995700 1 -- 172.21.15.79:0/179134587 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3704105460 msgr2=0x7f3704079b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:08:46.572 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:45.182+0000 7f370b995700 1 --2- 172.21.15.79:0/179134587 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3704105460 0x7f3704079b10 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f36f4011670 tx=0x7f36f4011b50 comp rx=0 tx=0).stop 2024-05-04T12:08:46.572 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:45.182+0000 7f370b995700 1 -- 172.21.15.79:0/179134587 shutdown_connections 2024-05-04T12:08:46.572 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:45.182+0000 7f370b995700 1 --2- 172.21.15.79:0/179134587 >> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] conn(0x7f36f0038370 0x7f36f003a830 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:08:46.572 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:45.182+0000 7f370b995700 1 --2- 172.21.15.79:0/179134587 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3704105460 0x7f3704079b10 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:08:46.572 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:45.182+0000 7f370b995700 1 -- 172.21.15.79:0/179134587 >> 172.21.15.79:0/179134587 conn(0x7f3704100a20 msgr2=0x7f3704102110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:08:46.572 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:45.182+0000 7f370b995700 1 -- 172.21.15.79:0/179134587 shutdown_connections 2024-05-04T12:08:46.572 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:45.182+0000 7f370b995700 1 -- 172.21.15.79:0/179134587 wait complete. 2024-05-04T12:08:46.572 INFO:teuthology.orchestra.run.smithi079.stdout:Deploying alertmanager service with default placement... 2024-05-04T12:08:47.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:08:47 smithi079 ceph-mon[27222]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-04T12:08:47.733 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout Scheduled alertmanager update... 2024-05-04T12:08:47.733 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:46.989+0000 7f3585b2a700 1 Processor -- start 2024-05-04T12:08:47.733 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:46.989+0000 7f3585b2a700 1 -- start start 2024-05-04T12:08:47.733 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:46.989+0000 7f3585b2a700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3580104810 0x7f3580104c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:08:47.734 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:46.989+0000 7f3585b2a700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3580105200 con 0x7f3580104810 2024-05-04T12:08:47.734 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:46.989+0000 7f357f7fe700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3580104810 0x7f3580104c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:08:47.734 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:46.989+0000 7f357f7fe700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3580104810 0x7f3580104c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:34320/0 (socket says 172.21.15.79:34320) 2024-05-04T12:08:47.734 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:46.989+0000 7f357f7fe700 1 -- 172.21.15.79:0/1423210465 learned_addr learned my addr 172.21.15.79:0/1423210465 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:08:47.734 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:46.989+0000 7f357f7fe700 1 -- 172.21.15.79:0/1423210465 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3580105340 con 0x7f3580104810 2024-05-04T12:08:47.734 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:46.990+0000 7f357f7fe700 1 --2- 172.21.15.79:0/1423210465 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3580104810 0x7f3580104c30 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f35700098a0 tx=0x7f3570009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bab059e0b4faeaba server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:08:47.734 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:46.990+0000 7f357effd700 1 -- 172.21.15.79:0/1423210465 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3570004180 con 0x7f3580104810 2024-05-04T12:08:47.734 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:46.990+0000 7f357effd700 1 -- 172.21.15.79:0/1423210465 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f35700042e0 con 0x7f3580104810 2024-05-04T12:08:47.734 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:46.990+0000 7f357effd700 1 -- 172.21.15.79:0/1423210465 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3570011af0 con 0x7f3580104810 2024-05-04T12:08:47.734 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:46.990+0000 7f3585b2a700 1 -- 172.21.15.79:0/1423210465 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3580104810 msgr2=0x7f3580104c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:08:47.734 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:46.990+0000 7f3585b2a700 1 --2- 172.21.15.79:0/1423210465 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3580104810 0x7f3580104c30 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f35700098a0 tx=0x7f3570009bb0 comp rx=0 tx=0).stop 2024-05-04T12:08:47.734 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:46.990+0000 7f3585b2a700 1 -- 172.21.15.79:0/1423210465 shutdown_connections 2024-05-04T12:08:47.734 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:46.990+0000 7f3585b2a700 1 --2- 172.21.15.79:0/1423210465 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3580104810 0x7f3580104c30 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:08:47.735 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:46.990+0000 7f3585b2a700 1 -- 172.21.15.79:0/1423210465 >> 172.21.15.79:0/1423210465 conn(0x7f3580100190 msgr2=0x7f35801025b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:08:47.735 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:46.990+0000 7f3585b2a700 1 -- 172.21.15.79:0/1423210465 shutdown_connections 2024-05-04T12:08:47.735 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:46.990+0000 7f3585b2a700 1 -- 172.21.15.79:0/1423210465 wait complete. 2024-05-04T12:08:47.735 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:46.991+0000 7f3585b2a700 1 Processor -- start 2024-05-04T12:08:47.735 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:46.991+0000 7f3585b2a700 1 -- start start 2024-05-04T12:08:47.735 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:46.991+0000 7f3585b2a700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3580104810 0x7f358019a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:08:47.735 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:46.991+0000 7f3585b2a700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f358019a770 con 0x7f3580104810 2024-05-04T12:08:47.735 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:46.991+0000 7f357f7fe700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3580104810 0x7f358019a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:08:47.735 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:46.991+0000 7f357f7fe700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3580104810 0x7f358019a230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:34326/0 (socket says 172.21.15.79:34326) 2024-05-04T12:08:47.735 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:46.992+0000 7f357f7fe700 1 -- 172.21.15.79:0/3106243662 learned_addr learned my addr 172.21.15.79:0/3106243662 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:08:47.735 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:46.992+0000 7f357f7fe700 1 -- 172.21.15.79:0/3106243662 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3570009550 con 0x7f3580104810 2024-05-04T12:08:47.735 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:46.992+0000 7f357f7fe700 1 --2- 172.21.15.79:0/3106243662 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3580104810 0x7f358019a230 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f3570011670 tx=0x7f3570011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:08:47.735 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:46.992+0000 7f357d7fa700 1 -- 172.21.15.79:0/3106243662 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f35700039f0 con 0x7f3580104810 2024-05-04T12:08:47.735 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:46.992+0000 7f357d7fa700 1 -- 172.21.15.79:0/3106243662 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f3570003b50 con 0x7f3580104810 2024-05-04T12:08:47.736 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:46.992+0000 7f3585b2a700 1 -- 172.21.15.79:0/3106243662 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f358019a970 con 0x7f3580104810 2024-05-04T12:08:47.736 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:46.992+0000 7f357d7fa700 1 -- 172.21.15.79:0/3106243662 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f35700247c0 con 0x7f3580104810 2024-05-04T12:08:47.736 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:46.992+0000 7f3585b2a700 1 -- 172.21.15.79:0/3106243662 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f358019ae10 con 0x7f3580104810 2024-05-04T12:08:47.736 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:46.993+0000 7f357d7fa700 1 -- 172.21.15.79:0/3106243662 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7f357002d3f0 con 0x7f3580104810 2024-05-04T12:08:47.736 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:46.993+0000 7f357d7fa700 1 --2- 172.21.15.79:0/3106243662 >> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] conn(0x7f356c038320 0x7f356c03a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:08:47.736 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:46.993+0000 7f3585b2a700 1 -- 172.21.15.79:0/3106243662 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35800623c0 con 0x7f3580104810 2024-05-04T12:08:47.736 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:46.993+0000 7f357d7fa700 1 -- 172.21.15.79:0/3106243662 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f357004c3a0 con 0x7f3580104810 2024-05-04T12:08:47.736 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:46.993+0000 7f3576ffd700 1 --2- 172.21.15.79:0/3106243662 >> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] conn(0x7f356c038320 0x7f356c03a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:08:47.736 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:46.994+0000 7f3576ffd700 1 --2- 172.21.15.79:0/3106243662 >> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] conn(0x7f356c038320 0x7f356c03a7e0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f3568006fd0 tx=0x7f3568006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:08:47.736 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:46.998+0000 7f357d7fa700 1 -- 172.21.15.79:0/3106243662 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f357002d6f0 con 0x7f3580104810 2024-05-04T12:08:47.736 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:47.196+0000 7f3585b2a700 1 -- 172.21.15.79:0/3106243662 --> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}) v1 -- 0x7f3580101650 con 0x7f356c038320 2024-05-04T12:08:47.736 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:47.201+0000 7f357d7fa700 1 -- 172.21.15.79:0/3106243662 <== mgr.14120 v2:172.21.15.79:6800/1620705054 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+33 (secure 0 0 0) 0x7f3580101650 con 0x7f356c038320 2024-05-04T12:08:47.736 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:47.203+0000 7f3585b2a700 1 -- 172.21.15.79:0/3106243662 >> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] conn(0x7f356c038320 msgr2=0x7f356c03a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:08:47.736 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:47.203+0000 7f3585b2a700 1 --2- 172.21.15.79:0/3106243662 >> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] conn(0x7f356c038320 0x7f356c03a7e0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f3568006fd0 tx=0x7f3568006e40 comp rx=0 tx=0).stop 2024-05-04T12:08:47.737 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:47.203+0000 7f3585b2a700 1 -- 172.21.15.79:0/3106243662 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3580104810 msgr2=0x7f358019a230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:08:47.737 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:47.203+0000 7f3585b2a700 1 --2- 172.21.15.79:0/3106243662 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3580104810 0x7f358019a230 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f3570011670 tx=0x7f3570011b50 comp rx=0 tx=0).stop 2024-05-04T12:08:47.737 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:47.203+0000 7f3585b2a700 1 -- 172.21.15.79:0/3106243662 shutdown_connections 2024-05-04T12:08:47.737 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:47.203+0000 7f3585b2a700 1 --2- 172.21.15.79:0/3106243662 >> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] conn(0x7f356c038320 0x7f356c03a7e0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:08:47.737 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:47.203+0000 7f3585b2a700 1 --2- 172.21.15.79:0/3106243662 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3580104810 0x7f358019a230 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:08:47.737 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:47.203+0000 7f3585b2a700 1 -- 172.21.15.79:0/3106243662 >> 172.21.15.79:0/3106243662 conn(0x7f3580100190 msgr2=0x7f3580100f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:08:47.737 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:47.203+0000 7f3585b2a700 1 -- 172.21.15.79:0/3106243662 shutdown_connections 2024-05-04T12:08:47.738 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:47.203+0000 7f3585b2a700 1 -- 172.21.15.79:0/3106243662 wait complete. 2024-05-04T12:08:48.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:08:48 smithi079 ceph-mon[27222]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2024-05-04T12:08:48.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:08:48 smithi079 ceph-mon[27222]: Saving service alertmanager spec with placement count:1 2024-05-04T12:08:48.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:08:48 smithi079 ceph-mon[27222]: from='mgr.14120 172.21.15.79:0/2394226123' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:08:48.956 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:48.136+0000 7ff652a2b700 1 Processor -- start 2024-05-04T12:08:48.956 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:48.136+0000 7ff652a2b700 1 -- start start 2024-05-04T12:08:48.956 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:48.136+0000 7ff652a2b700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff64c104860 0x7ff64c104c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:08:48.956 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:48.136+0000 7ff652a2b700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff64c105250 con 0x7ff64c104860 2024-05-04T12:08:48.956 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:48.137+0000 7ff64bfff700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff64c104860 0x7ff64c104c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:08:48.956 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:48.137+0000 7ff64bfff700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff64c104860 0x7ff64c104c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:34338/0 (socket says 172.21.15.79:34338) 2024-05-04T12:08:48.957 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:48.137+0000 7ff64bfff700 1 -- 172.21.15.79:0/4280263901 learned_addr learned my addr 172.21.15.79:0/4280263901 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:08:48.957 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:48.137+0000 7ff64bfff700 1 -- 172.21.15.79:0/4280263901 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff64c1077f0 con 0x7ff64c104860 2024-05-04T12:08:48.957 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:48.137+0000 7ff64bfff700 1 --2- 172.21.15.79:0/4280263901 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff64c104860 0x7ff64c104c80 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ff63c0098a0 tx=0x7ff63c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a017a86a622977b server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:08:48.957 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:48.137+0000 7ff64affd700 1 -- 172.21.15.79:0/4280263901 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff63c004180 con 0x7ff64c104860 2024-05-04T12:08:48.957 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:48.137+0000 7ff64affd700 1 -- 172.21.15.79:0/4280263901 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7ff63c0042e0 con 0x7ff64c104860 2024-05-04T12:08:48.957 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:48.137+0000 7ff64affd700 1 -- 172.21.15.79:0/4280263901 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff63c011af0 con 0x7ff64c104860 2024-05-04T12:08:48.957 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:48.138+0000 7ff652a2b700 1 -- 172.21.15.79:0/4280263901 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff64c104860 msgr2=0x7ff64c104c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:08:48.957 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:48.138+0000 7ff652a2b700 1 --2- 172.21.15.79:0/4280263901 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff64c104860 0x7ff64c104c80 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ff63c0098a0 tx=0x7ff63c009bb0 comp rx=0 tx=0).stop 2024-05-04T12:08:48.957 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:48.138+0000 7ff652a2b700 1 -- 172.21.15.79:0/4280263901 shutdown_connections 2024-05-04T12:08:48.957 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:48.138+0000 7ff652a2b700 1 --2- 172.21.15.79:0/4280263901 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff64c104860 0x7ff64c104c80 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:08:48.957 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:48.138+0000 7ff652a2b700 1 -- 172.21.15.79:0/4280263901 >> 172.21.15.79:0/4280263901 conn(0x7ff64c100190 msgr2=0x7ff64c1025d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:08:48.957 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:48.138+0000 7ff652a2b700 1 -- 172.21.15.79:0/4280263901 shutdown_connections 2024-05-04T12:08:48.957 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:48.138+0000 7ff652a2b700 1 -- 172.21.15.79:0/4280263901 wait complete. 2024-05-04T12:08:48.957 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:48.138+0000 7ff652a2b700 1 Processor -- start 2024-05-04T12:08:48.958 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:48.138+0000 7ff652a2b700 1 -- start start 2024-05-04T12:08:48.958 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:48.139+0000 7ff652a2b700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff64c104860 0x7ff64c107150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:08:48.958 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:48.139+0000 7ff652a2b700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff64c107690 con 0x7ff64c104860 2024-05-04T12:08:48.958 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:48.139+0000 7ff64bfff700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff64c104860 0x7ff64c107150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:08:48.958 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:48.139+0000 7ff64bfff700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff64c104860 0x7ff64c107150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:34348/0 (socket says 172.21.15.79:34348) 2024-05-04T12:08:48.958 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:48.139+0000 7ff64bfff700 1 -- 172.21.15.79:0/1419046409 learned_addr learned my addr 172.21.15.79:0/1419046409 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:08:48.958 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:48.139+0000 7ff64bfff700 1 -- 172.21.15.79:0/1419046409 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff63c009550 con 0x7ff64c104860 2024-05-04T12:08:48.958 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:48.139+0000 7ff64bfff700 1 --2- 172.21.15.79:0/1419046409 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff64c104860 0x7ff64c107150 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ff63c000c00 tx=0x7ff63c011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:08:48.958 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:48.139+0000 7ff6497fa700 1 -- 172.21.15.79:0/1419046409 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff63c003a70 con 0x7ff64c104860 2024-05-04T12:08:48.958 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:48.139+0000 7ff652a2b700 1 -- 172.21.15.79:0/1419046409 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff64c1057a0 con 0x7ff64c104860 2024-05-04T12:08:48.958 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:48.139+0000 7ff6497fa700 1 -- 172.21.15.79:0/1419046409 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7ff63c003bd0 con 0x7ff64c104860 2024-05-04T12:08:48.958 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:48.139+0000 7ff6497fa700 1 -- 172.21.15.79:0/1419046409 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff63c0238c0 con 0x7ff64c104860 2024-05-04T12:08:48.958 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:48.139+0000 7ff652a2b700 1 -- 172.21.15.79:0/1419046409 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff64c105c40 con 0x7ff64c104860 2024-05-04T12:08:48.959 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:48.140+0000 7ff652a2b700 1 -- 172.21.15.79:0/1419046409 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff64c0623c0 con 0x7ff64c104860 2024-05-04T12:08:48.959 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:48.140+0000 7ff6497fa700 1 -- 172.21.15.79:0/1419046409 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7ff63c023b80 con 0x7ff64c104860 2024-05-04T12:08:48.959 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:48.141+0000 7ff6497fa700 1 --2- 172.21.15.79:0/1419046409 >> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] conn(0x7ff634038320 0x7ff63403a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:08:48.959 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:48.141+0000 7ff6497fa700 1 -- 172.21.15.79:0/1419046409 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7ff63c04c1d0 con 0x7ff64c104860 2024-05-04T12:08:48.959 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:48.141+0000 7ff64b7fe700 1 --2- 172.21.15.79:0/1419046409 >> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] conn(0x7ff634038320 0x7ff63403a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:08:48.959 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:48.141+0000 7ff64b7fe700 1 --2- 172.21.15.79:0/1419046409 >> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] conn(0x7ff634038320 0x7ff63403a7e0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7ff640006fd0 tx=0x7ff640006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:08:48.959 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:48.145+0000 7ff6497fa700 1 -- 172.21.15.79:0/1419046409 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7ff63c0193c0 con 0x7ff64c104860 2024-05-04T12:08:48.959 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:48.332+0000 7ff652a2b700 1 -- 172.21.15.79:0/1419046409 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7ff64c10aa20 con 0x7ff64c104860 2024-05-04T12:08:48.959 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:48.335+0000 7ff6497fa700 1 -- 172.21.15.79:0/1419046409 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (secure 0 0 0) 0x7ff63c010380 con 0x7ff64c104860 2024-05-04T12:08:48.959 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:48.338+0000 7ff652a2b700 1 -- 172.21.15.79:0/1419046409 >> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] conn(0x7ff634038320 msgr2=0x7ff63403a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:08:48.959 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:48.338+0000 7ff652a2b700 1 --2- 172.21.15.79:0/1419046409 >> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] conn(0x7ff634038320 0x7ff63403a7e0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7ff640006fd0 tx=0x7ff640006e40 comp rx=0 tx=0).stop 2024-05-04T12:08:48.959 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:48.338+0000 7ff652a2b700 1 -- 172.21.15.79:0/1419046409 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff64c104860 msgr2=0x7ff64c107150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:08:48.959 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:48.338+0000 7ff652a2b700 1 --2- 172.21.15.79:0/1419046409 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff64c104860 0x7ff64c107150 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ff63c000c00 tx=0x7ff63c011b50 comp rx=0 tx=0).stop 2024-05-04T12:08:48.959 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:48.338+0000 7ff652a2b700 1 -- 172.21.15.79:0/1419046409 shutdown_connections 2024-05-04T12:08:48.960 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:48.338+0000 7ff652a2b700 1 --2- 172.21.15.79:0/1419046409 >> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] conn(0x7ff634038320 0x7ff63403a7e0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:08:48.960 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:48.338+0000 7ff652a2b700 1 --2- 172.21.15.79:0/1419046409 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff64c104860 0x7ff64c107150 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:08:48.960 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:48.338+0000 7ff652a2b700 1 -- 172.21.15.79:0/1419046409 >> 172.21.15.79:0/1419046409 conn(0x7ff64c100190 msgr2=0x7ff64c100f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:08:48.960 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:48.338+0000 7ff652a2b700 1 -- 172.21.15.79:0/1419046409 shutdown_connections 2024-05-04T12:08:48.960 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:48.338+0000 7ff652a2b700 1 -- 172.21.15.79:0/1419046409 wait complete. 2024-05-04T12:08:49.538 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:08:49 smithi079 ceph-mon[27222]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-04T12:08:49.538 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:08:49 smithi079 ceph-mon[27222]: from='client.? 172.21.15.79:0/1419046409' entity='client.admin' 2024-05-04T12:08:50.121 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:49.346+0000 7f44af144700 1 Processor -- start 2024-05-04T12:08:50.121 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:49.346+0000 7f44af144700 1 -- start start 2024-05-04T12:08:50.121 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:49.346+0000 7f44af144700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f44a8107030 0x7f44a8109420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:08:50.121 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:49.346+0000 7f44af144700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44a8079e30 con 0x7f44a8107030 2024-05-04T12:08:50.121 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:49.346+0000 7f44acee0700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f44a8107030 0x7f44a8109420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:08:50.121 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:49.346+0000 7f44acee0700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f44a8107030 0x7f44a8109420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:34358/0 (socket says 172.21.15.79:34358) 2024-05-04T12:08:50.121 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:49.346+0000 7f44acee0700 1 -- 172.21.15.79:0/2220095373 learned_addr learned my addr 172.21.15.79:0/2220095373 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:08:50.121 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:49.346+0000 7f44acee0700 1 -- 172.21.15.79:0/2220095373 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44a8109960 con 0x7f44a8107030 2024-05-04T12:08:50.121 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:49.346+0000 7f44acee0700 1 --2- 172.21.15.79:0/2220095373 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f44a8107030 0x7f44a8109420 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f449c0098a0 tx=0x7f449c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c83f2a8c350fec1 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:08:50.121 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:49.346+0000 7f44a77fe700 1 -- 172.21.15.79:0/2220095373 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f449c004180 con 0x7f44a8107030 2024-05-04T12:08:50.121 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:49.347+0000 7f44a77fe700 1 -- 172.21.15.79:0/2220095373 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f449c0042e0 con 0x7f44a8107030 2024-05-04T12:08:50.121 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:49.347+0000 7f44a77fe700 1 -- 172.21.15.79:0/2220095373 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f449c011af0 con 0x7f44a8107030 2024-05-04T12:08:50.122 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:49.347+0000 7f44af144700 1 -- 172.21.15.79:0/2220095373 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f44a8107030 msgr2=0x7f44a8109420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:08:50.122 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:49.347+0000 7f44af144700 1 --2- 172.21.15.79:0/2220095373 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f44a8107030 0x7f44a8109420 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f449c0098a0 tx=0x7f449c009bb0 comp rx=0 tx=0).stop 2024-05-04T12:08:50.122 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:49.347+0000 7f44af144700 1 -- 172.21.15.79:0/2220095373 shutdown_connections 2024-05-04T12:08:50.122 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:49.347+0000 7f44af144700 1 --2- 172.21.15.79:0/2220095373 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f44a8107030 0x7f44a8109420 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:08:50.122 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:49.347+0000 7f44af144700 1 -- 172.21.15.79:0/2220095373 >> 172.21.15.79:0/2220095373 conn(0x7f44a8100a20 msgr2=0x7f44a8102e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:08:50.122 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:49.347+0000 7f44af144700 1 -- 172.21.15.79:0/2220095373 shutdown_connections 2024-05-04T12:08:50.122 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:49.347+0000 7f44af144700 1 -- 172.21.15.79:0/2220095373 wait complete. 2024-05-04T12:08:50.122 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:49.348+0000 7f44af144700 1 Processor -- start 2024-05-04T12:08:50.122 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:49.348+0000 7f44af144700 1 -- start start 2024-05-04T12:08:50.122 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:49.348+0000 7f44af144700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f44a8107030 0x7f44a8106990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:08:50.122 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:49.348+0000 7f44af144700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44a8106ed0 con 0x7f44a8107030 2024-05-04T12:08:50.122 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:49.348+0000 7f44acee0700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f44a8107030 0x7f44a8106990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:08:50.122 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:49.348+0000 7f44acee0700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f44a8107030 0x7f44a8106990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:34374/0 (socket says 172.21.15.79:34374) 2024-05-04T12:08:50.122 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:49.348+0000 7f44acee0700 1 -- 172.21.15.79:0/2760134196 learned_addr learned my addr 172.21.15.79:0/2760134196 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:08:50.123 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:49.348+0000 7f44acee0700 1 -- 172.21.15.79:0/2760134196 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f449c009550 con 0x7f44a8107030 2024-05-04T12:08:50.123 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:49.348+0000 7f44acee0700 1 --2- 172.21.15.79:0/2760134196 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f44a8107030 0x7f44a8106990 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f449c011670 tx=0x7f449c011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:08:50.123 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:49.348+0000 7f44a5ffb700 1 -- 172.21.15.79:0/2760134196 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f449c003a40 con 0x7f44a8107030 2024-05-04T12:08:50.123 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:49.348+0000 7f44a5ffb700 1 -- 172.21.15.79:0/2760134196 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f449c003ba0 con 0x7f44a8107030 2024-05-04T12:08:50.123 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:49.348+0000 7f44af144700 1 -- 172.21.15.79:0/2760134196 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44a8104fe0 con 0x7f44a8107030 2024-05-04T12:08:50.123 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:49.350+0000 7f44a5ffb700 1 -- 172.21.15.79:0/2760134196 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f449c024790 con 0x7f44a8107030 2024-05-04T12:08:50.123 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:49.350+0000 7f44af144700 1 -- 172.21.15.79:0/2760134196 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44a8105480 con 0x7f44a8107030 2024-05-04T12:08:50.123 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:49.350+0000 7f44a5ffb700 1 -- 172.21.15.79:0/2760134196 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7f449c02d3f0 con 0x7f44a8107030 2024-05-04T12:08:50.123 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:49.350+0000 7f44a5ffb700 1 --2- 172.21.15.79:0/2760134196 >> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] conn(0x7f4498038320 0x7f449803a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:08:50.123 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:49.350+0000 7f44af144700 1 -- 172.21.15.79:0/2760134196 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44a810a6f0 con 0x7f44a8107030 2024-05-04T12:08:50.123 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:49.350+0000 7f44a5ffb700 1 -- 172.21.15.79:0/2760134196 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f449c04c6f0 con 0x7f44a8107030 2024-05-04T12:08:50.124 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:49.350+0000 7f44a7fff700 1 --2- 172.21.15.79:0/2760134196 >> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] conn(0x7f4498038320 0x7f449803a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:08:50.124 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:49.350+0000 7f44a7fff700 1 --2- 172.21.15.79:0/2760134196 >> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] conn(0x7f4498038320 0x7f449803a7e0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f4494006fd0 tx=0x7f4494006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:08:50.124 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:49.354+0000 7f44a5ffb700 1 -- 172.21.15.79:0/2760134196 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f449c010970 con 0x7f44a8107030 2024-05-04T12:08:50.124 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:49.540+0000 7f44af144700 1 -- 172.21.15.79:0/2760134196 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7f44a8105ae0 con 0x7f44a8107030 2024-05-04T12:08:50.124 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:49.543+0000 7f44a5ffb700 1 -- 172.21.15.79:0/2760134196 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (secure 0 0 0) 0x7f449c008040 con 0x7f44a8107030 2024-05-04T12:08:50.124 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:49.545+0000 7f44af144700 1 -- 172.21.15.79:0/2760134196 >> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] conn(0x7f4498038320 msgr2=0x7f449803a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:08:50.124 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:49.545+0000 7f44af144700 1 --2- 172.21.15.79:0/2760134196 >> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] conn(0x7f4498038320 0x7f449803a7e0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f4494006fd0 tx=0x7f4494006e40 comp rx=0 tx=0).stop 2024-05-04T12:08:50.124 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:49.545+0000 7f44af144700 1 -- 172.21.15.79:0/2760134196 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f44a8107030 msgr2=0x7f44a8106990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:08:50.124 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:49.545+0000 7f44af144700 1 --2- 172.21.15.79:0/2760134196 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f44a8107030 0x7f44a8106990 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f449c011670 tx=0x7f449c011b50 comp rx=0 tx=0).stop 2024-05-04T12:08:50.124 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:49.545+0000 7f44af144700 1 -- 172.21.15.79:0/2760134196 shutdown_connections 2024-05-04T12:08:50.124 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:49.545+0000 7f44af144700 1 --2- 172.21.15.79:0/2760134196 >> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] conn(0x7f4498038320 0x7f449803a7e0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:08:50.125 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:49.546+0000 7f44af144700 1 --2- 172.21.15.79:0/2760134196 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f44a8107030 0x7f44a8106990 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:08:50.125 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:49.546+0000 7f44af144700 1 -- 172.21.15.79:0/2760134196 >> 172.21.15.79:0/2760134196 conn(0x7f44a8100a20 msgr2=0x7f44a81017b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:08:50.125 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:49.546+0000 7f44af144700 1 -- 172.21.15.79:0/2760134196 shutdown_connections 2024-05-04T12:08:50.125 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:49.546+0000 7f44af144700 1 -- 172.21.15.79:0/2760134196 wait complete. 2024-05-04T12:08:50.125 INFO:teuthology.orchestra.run.smithi079.stdout:Enabling the dashboard module... 2024-05-04T12:08:50.695 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:08:50 smithi079 ceph-mon[27222]: from='client.? 172.21.15.79:0/2760134196' entity='client.admin' 2024-05-04T12:08:51.868 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:08:51 smithi079 ceph-mon[27222]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-04T12:08:51.869 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:08:51 smithi079 ceph-mon[27222]: from='client.? 172.21.15.79:0/928257167' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-05-04T12:08:52.197 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:50.529+0000 7f0692c7d700 1 Processor -- start 2024-05-04T12:08:52.203 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:50.529+0000 7f0692c7d700 1 -- start start 2024-05-04T12:08:52.203 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:50.529+0000 7f0692c7d700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f068c105950 0x7f068c105d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:08:52.203 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:50.529+0000 7f0692c7d700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f068c106340 con 0x7f068c105950 2024-05-04T12:08:52.203 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:50.529+0000 7f0690a19700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f068c105950 0x7f068c105d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:08:52.203 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:50.529+0000 7f0690a19700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f068c105950 0x7f068c105d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:34380/0 (socket says 172.21.15.79:34380) 2024-05-04T12:08:52.203 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:50.529+0000 7f0690a19700 1 -- 172.21.15.79:0/800241004 learned_addr learned my addr 172.21.15.79:0/800241004 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:08:52.203 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:50.530+0000 7f0690a19700 1 -- 172.21.15.79:0/800241004 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f068c106670 con 0x7f068c105950 2024-05-04T12:08:52.204 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:50.530+0000 7f0690a19700 1 --2- 172.21.15.79:0/800241004 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f068c105950 0x7f068c105d70 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f06800098a0 tx=0x7f0680009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18e48c48a5da434a server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:08:52.204 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:50.530+0000 7f068b7fe700 1 -- 172.21.15.79:0/800241004 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0680004180 con 0x7f068c105950 2024-05-04T12:08:52.204 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:50.530+0000 7f068b7fe700 1 -- 172.21.15.79:0/800241004 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f06800042e0 con 0x7f068c105950 2024-05-04T12:08:52.204 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:50.530+0000 7f068b7fe700 1 -- 172.21.15.79:0/800241004 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0680011af0 con 0x7f068c105950 2024-05-04T12:08:52.204 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:50.530+0000 7f0692c7d700 1 -- 172.21.15.79:0/800241004 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f068c105950 msgr2=0x7f068c105d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:08:52.204 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:50.530+0000 7f0692c7d700 1 --2- 172.21.15.79:0/800241004 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f068c105950 0x7f068c105d70 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f06800098a0 tx=0x7f0680009bb0 comp rx=0 tx=0).stop 2024-05-04T12:08:52.204 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:50.530+0000 7f0692c7d700 1 -- 172.21.15.79:0/800241004 shutdown_connections 2024-05-04T12:08:52.204 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:50.530+0000 7f0692c7d700 1 --2- 172.21.15.79:0/800241004 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f068c105950 0x7f068c105d70 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:08:52.204 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:50.530+0000 7f0692c7d700 1 -- 172.21.15.79:0/800241004 >> 172.21.15.79:0/800241004 conn(0x7f068c100170 msgr2=0x7f068c1025b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:08:52.204 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:50.531+0000 7f0692c7d700 1 -- 172.21.15.79:0/800241004 shutdown_connections 2024-05-04T12:08:52.204 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:50.531+0000 7f0692c7d700 1 -- 172.21.15.79:0/800241004 wait complete. 2024-05-04T12:08:52.204 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:50.531+0000 7f0692c7d700 1 Processor -- start 2024-05-04T12:08:52.204 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:50.531+0000 7f0692c7d700 1 -- start start 2024-05-04T12:08:52.205 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:50.531+0000 7f0692c7d700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f068c105950 0x7f068c105070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:08:52.205 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:50.531+0000 7f0692c7d700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f068c19cef0 con 0x7f068c105950 2024-05-04T12:08:52.205 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:50.531+0000 7f0690a19700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f068c105950 0x7f068c105070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:08:52.205 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:50.531+0000 7f0690a19700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f068c105950 0x7f068c105070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:34394/0 (socket says 172.21.15.79:34394) 2024-05-04T12:08:52.205 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:50.531+0000 7f0690a19700 1 -- 172.21.15.79:0/928257167 learned_addr learned my addr 172.21.15.79:0/928257167 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:08:52.205 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:50.532+0000 7f0690a19700 1 -- 172.21.15.79:0/928257167 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0680009550 con 0x7f068c105950 2024-05-04T12:08:52.205 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:50.532+0000 7f0690a19700 1 --2- 172.21.15.79:0/928257167 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f068c105950 0x7f068c105070 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f0680011670 tx=0x7f0680011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:08:52.206 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:50.532+0000 7f0689ffb700 1 -- 172.21.15.79:0/928257167 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0680003a40 con 0x7f068c105950 2024-05-04T12:08:52.206 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:50.532+0000 7f0689ffb700 1 -- 172.21.15.79:0/928257167 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f0680003ba0 con 0x7f068c105950 2024-05-04T12:08:52.206 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:50.532+0000 7f0692c7d700 1 -- 172.21.15.79:0/928257167 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f068c19d030 con 0x7f068c105950 2024-05-04T12:08:52.206 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:50.532+0000 7f0689ffb700 1 -- 172.21.15.79:0/928257167 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0680024790 con 0x7f068c105950 2024-05-04T12:08:52.206 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:50.532+0000 7f0692c7d700 1 -- 172.21.15.79:0/928257167 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f068c19d4d0 con 0x7f068c105950 2024-05-04T12:08:52.206 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:50.533+0000 7f0689ffb700 1 -- 172.21.15.79:0/928257167 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7f068001a430 con 0x7f068c105950 2024-05-04T12:08:52.206 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:50.533+0000 7f0692c7d700 1 -- 172.21.15.79:0/928257167 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f068c0623c0 con 0x7f068c105950 2024-05-04T12:08:52.206 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:50.533+0000 7f0689ffb700 1 --2- 172.21.15.79:0/928257167 >> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] conn(0x7f067c038370 0x7f067c03a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:08:52.206 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:50.533+0000 7f0689ffb700 1 -- 172.21.15.79:0/928257167 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f068004c4b0 con 0x7f068c105950 2024-05-04T12:08:52.206 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:50.534+0000 7f068bfff700 1 --2- 172.21.15.79:0/928257167 >> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] conn(0x7f067c038370 0x7f067c03a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:08:52.206 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:50.534+0000 7f068bfff700 1 --2- 172.21.15.79:0/928257167 >> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] conn(0x7f067c038370 0x7f067c03a830 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f0678006fd0 tx=0x7f0678006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:08:52.206 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:50.538+0000 7f0689ffb700 1 -- 172.21.15.79:0/928257167 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f068001ac50 con 0x7f068c105950 2024-05-04T12:08:52.207 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:50.787+0000 7f0692c7d700 1 -- 172.21.15.79:0/928257167 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7f068c19dbf0 con 0x7f068c105950 2024-05-04T12:08:52.207 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:51.546+0000 7f0689ffb700 1 -- 172.21.15.79:0/928257167 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mgrmap(e 9) v1 ==== 45296+0+0 (secure 0 0 0) 0x7f068001a7e0 con 0x7f068c105950 2024-05-04T12:08:52.207 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:51.548+0000 7f0689ffb700 1 -- 172.21.15.79:0/928257167 <== mon.0 v2:172.21.15.79:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7f0680010820 con 0x7f068c105950 2024-05-04T12:08:52.207 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:51.551+0000 7f0692c7d700 1 -- 172.21.15.79:0/928257167 >> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] conn(0x7f067c038370 msgr2=0x7f067c03a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:08:52.207 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:51.551+0000 7f0692c7d700 1 --2- 172.21.15.79:0/928257167 >> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] conn(0x7f067c038370 0x7f067c03a830 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f0678006fd0 tx=0x7f0678006e40 comp rx=0 tx=0).stop 2024-05-04T12:08:52.207 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:51.551+0000 7f0692c7d700 1 -- 172.21.15.79:0/928257167 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f068c105950 msgr2=0x7f068c105070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:08:52.207 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:51.551+0000 7f0692c7d700 1 --2- 172.21.15.79:0/928257167 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f068c105950 0x7f068c105070 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f0680011670 tx=0x7f0680011b50 comp rx=0 tx=0).stop 2024-05-04T12:08:52.207 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:51.551+0000 7f0692c7d700 1 -- 172.21.15.79:0/928257167 shutdown_connections 2024-05-04T12:08:52.207 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:51.552+0000 7f0692c7d700 1 --2- 172.21.15.79:0/928257167 >> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] conn(0x7f067c038370 0x7f067c03a830 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:08:52.207 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:51.552+0000 7f0692c7d700 1 --2- 172.21.15.79:0/928257167 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f068c105950 0x7f068c105070 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:08:52.207 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:51.552+0000 7f0692c7d700 1 -- 172.21.15.79:0/928257167 >> 172.21.15.79:0/928257167 conn(0x7f068c100170 msgr2=0x7f068c100f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:08:52.207 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:51.552+0000 7f0692c7d700 1 -- 172.21.15.79:0/928257167 shutdown_connections 2024-05-04T12:08:52.207 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:51.552+0000 7f0692c7d700 1 -- 172.21.15.79:0/928257167 wait complete. 2024-05-04T12:08:52.821 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:08:52 smithi079 ceph-mon[27222]: from='client.? 172.21.15.79:0/928257167' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-05-04T12:08:52.821 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:08:52 smithi079 ceph-mon[27222]: mgrmap e9: smithi079.dgcpwl(active, since 25s) 2024-05-04T12:08:53.476 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout { 2024-05-04T12:08:53.476 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-05-04T12:08:53.476 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "available": true, 2024-05-04T12:08:53.476 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "active_name": "smithi079.dgcpwl", 2024-05-04T12:08:53.476 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-05-04T12:08:53.476 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout } 2024-05-04T12:08:53.476 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:52.617+0000 7fd7ceb53700 1 Processor -- start 2024-05-04T12:08:53.476 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:52.617+0000 7fd7ceb53700 1 -- start start 2024-05-04T12:08:53.477 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:52.617+0000 7fd7ceb53700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd7c8106560 0x7fd7c8106980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:08:53.477 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:52.617+0000 7fd7ceb53700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7c8106f50 con 0x7fd7c8106560 2024-05-04T12:08:53.477 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:52.617+0000 7fd7cc8ef700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd7c8106560 0x7fd7c8106980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:08:53.477 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:52.617+0000 7fd7cc8ef700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd7c8106560 0x7fd7c8106980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:34424/0 (socket says 172.21.15.79:34424) 2024-05-04T12:08:53.477 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:52.617+0000 7fd7cc8ef700 1 -- 172.21.15.79:0/1521691006 learned_addr learned my addr 172.21.15.79:0/1521691006 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:08:53.477 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:52.617+0000 7fd7cc8ef700 1 -- 172.21.15.79:0/1521691006 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7c8107760 con 0x7fd7c8106560 2024-05-04T12:08:53.477 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:52.618+0000 7fd7cc8ef700 1 --2- 172.21.15.79:0/1521691006 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd7c8106560 0x7fd7c8106980 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fd7bc0098a0 tx=0x7fd7bc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=616782f59ff1279f server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:08:53.477 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:52.618+0000 7fd7c77fe700 1 -- 172.21.15.79:0/1521691006 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd7bc004180 con 0x7fd7c8106560 2024-05-04T12:08:53.477 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:52.618+0000 7fd7c77fe700 1 -- 172.21.15.79:0/1521691006 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fd7bc0042e0 con 0x7fd7c8106560 2024-05-04T12:08:53.477 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:52.618+0000 7fd7c77fe700 1 -- 172.21.15.79:0/1521691006 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd7bc011af0 con 0x7fd7c8106560 2024-05-04T12:08:53.477 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:52.619+0000 7fd7ceb53700 1 -- 172.21.15.79:0/1521691006 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd7c8106560 msgr2=0x7fd7c8106980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:08:53.477 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:52.619+0000 7fd7ceb53700 1 --2- 172.21.15.79:0/1521691006 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd7c8106560 0x7fd7c8106980 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fd7bc0098a0 tx=0x7fd7bc009bb0 comp rx=0 tx=0).stop 2024-05-04T12:08:53.477 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:52.619+0000 7fd7ceb53700 1 -- 172.21.15.79:0/1521691006 shutdown_connections 2024-05-04T12:08:53.477 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:52.619+0000 7fd7ceb53700 1 --2- 172.21.15.79:0/1521691006 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd7c8106560 0x7fd7c8106980 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:08:53.478 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:52.619+0000 7fd7ceb53700 1 -- 172.21.15.79:0/1521691006 >> 172.21.15.79:0/1521691006 conn(0x7fd7c8101b40 msgr2=0x7fd7c8103f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:08:53.478 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:52.619+0000 7fd7ceb53700 1 -- 172.21.15.79:0/1521691006 shutdown_connections 2024-05-04T12:08:53.478 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:52.619+0000 7fd7ceb53700 1 -- 172.21.15.79:0/1521691006 wait complete. 2024-05-04T12:08:53.478 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:52.619+0000 7fd7ceb53700 1 Processor -- start 2024-05-04T12:08:53.478 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:52.619+0000 7fd7ceb53700 1 -- start start 2024-05-04T12:08:53.478 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:52.619+0000 7fd7ceb53700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd7c8106560 0x7fd7c819a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:08:53.478 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:52.619+0000 7fd7ceb53700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7c819a730 con 0x7fd7c8106560 2024-05-04T12:08:53.478 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:52.620+0000 7fd7cc8ef700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd7c8106560 0x7fd7c819a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:08:53.478 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:52.620+0000 7fd7cc8ef700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd7c8106560 0x7fd7c819a1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:34440/0 (socket says 172.21.15.79:34440) 2024-05-04T12:08:53.478 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:52.620+0000 7fd7cc8ef700 1 -- 172.21.15.79:0/568999833 learned_addr learned my addr 172.21.15.79:0/568999833 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:08:53.478 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:52.620+0000 7fd7cc8ef700 1 -- 172.21.15.79:0/568999833 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7bc009550 con 0x7fd7c8106560 2024-05-04T12:08:53.478 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:52.620+0000 7fd7cc8ef700 1 --2- 172.21.15.79:0/568999833 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd7c8106560 0x7fd7c819a1f0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd7bc006ba0 tx=0x7fd7bc011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:08:53.478 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:52.620+0000 7fd7c5ffb700 1 -- 172.21.15.79:0/568999833 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd7bc003b00 con 0x7fd7c8106560 2024-05-04T12:08:53.479 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:52.620+0000 7fd7c5ffb700 1 -- 172.21.15.79:0/568999833 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fd7bc01a8e0 con 0x7fd7c8106560 2024-05-04T12:08:53.479 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:52.620+0000 7fd7c5ffb700 1 -- 172.21.15.79:0/568999833 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd7bc019990 con 0x7fd7c8106560 2024-05-04T12:08:53.479 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:52.620+0000 7fd7ceb53700 1 -- 172.21.15.79:0/568999833 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7c819a930 con 0x7fd7c8106560 2024-05-04T12:08:53.479 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:52.621+0000 7fd7ceb53700 1 -- 172.21.15.79:0/568999833 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7c819add0 con 0x7fd7c8106560 2024-05-04T12:08:53.479 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:52.621+0000 7fd7c5ffb700 1 -- 172.21.15.79:0/568999833 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 9) v1 ==== 45296+0+0 (secure 0 0 0) 0x7fd7bc019c50 con 0x7fd7c8106560 2024-05-04T12:08:53.479 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:52.621+0000 7fd7ceb53700 1 -- 172.21.15.79:0/568999833 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7c80623c0 con 0x7fd7c8106560 2024-05-04T12:08:53.479 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:52.622+0000 7fd7c5ffb700 1 --2- 172.21.15.79:0/568999833 >> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] conn(0x7fd7b8038320 0x7fd7b803a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:08:53.479 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:52.622+0000 7fd7c5ffb700 1 -- 172.21.15.79:0/568999833 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fd7bc04c6d0 con 0x7fd7c8106560 2024-05-04T12:08:53.479 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:52.622+0000 7fd7c7fff700 1 -- 172.21.15.79:0/568999833 >> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] conn(0x7fd7b8038320 msgr2=0x7fd7b803a7e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.79:6800/1620705054 2024-05-04T12:08:53.479 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:52.622+0000 7fd7c7fff700 1 --2- 172.21.15.79:0/568999833 >> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] conn(0x7fd7b8038320 0x7fd7b803a7e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-04T12:08:53.479 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:52.626+0000 7fd7c5ffb700 1 -- 172.21.15.79:0/568999833 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fd7bc018710 con 0x7fd7c8106560 2024-05-04T12:08:53.479 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:52.823+0000 7fd7c7fff700 1 -- 172.21.15.79:0/568999833 >> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] conn(0x7fd7b8038320 msgr2=0x7fd7b803a7e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.79:6800/1620705054 2024-05-04T12:08:53.479 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:52.823+0000 7fd7c7fff700 1 --2- 172.21.15.79:0/568999833 >> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] conn(0x7fd7b8038320 0x7fd7b803a7e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-04T12:08:53.479 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:52.911+0000 7fd7ceb53700 1 -- 172.21.15.79:0/568999833 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7fd7c810a990 con 0x7fd7c8106560 2024-05-04T12:08:53.480 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:52.911+0000 7fd7c5ffb700 1 -- 172.21.15.79:0/568999833 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+103 (secure 0 0 0) 0x7fd7bc018350 con 0x7fd7c8106560 2024-05-04T12:08:53.480 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:52.913+0000 7fd7ceb53700 1 -- 172.21.15.79:0/568999833 >> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] conn(0x7fd7b8038320 msgr2=0x7fd7b803a7e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-04T12:08:53.480 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:52.913+0000 7fd7ceb53700 1 --2- 172.21.15.79:0/568999833 >> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] conn(0x7fd7b8038320 0x7fd7b803a7e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:08:53.480 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:52.913+0000 7fd7ceb53700 1 -- 172.21.15.79:0/568999833 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd7c8106560 msgr2=0x7fd7c819a1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:08:53.480 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:52.913+0000 7fd7ceb53700 1 --2- 172.21.15.79:0/568999833 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd7c8106560 0x7fd7c819a1f0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd7bc006ba0 tx=0x7fd7bc011b50 comp rx=0 tx=0).stop 2024-05-04T12:08:53.480 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:52.913+0000 7fd7ceb53700 1 -- 172.21.15.79:0/568999833 shutdown_connections 2024-05-04T12:08:53.480 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:52.913+0000 7fd7ceb53700 1 --2- 172.21.15.79:0/568999833 >> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] conn(0x7fd7b8038320 0x7fd7b803a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:08:53.480 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:52.913+0000 7fd7ceb53700 1 --2- 172.21.15.79:0/568999833 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd7c8106560 0x7fd7c819a1f0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:08:53.480 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:52.913+0000 7fd7ceb53700 1 -- 172.21.15.79:0/568999833 >> 172.21.15.79:0/568999833 conn(0x7fd7c8101b40 msgr2=0x7fd7c8102890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:08:53.480 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:52.914+0000 7fd7ceb53700 1 -- 172.21.15.79:0/568999833 shutdown_connections 2024-05-04T12:08:53.480 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:52.914+0000 7fd7ceb53700 1 -- 172.21.15.79:0/568999833 wait complete. 2024-05-04T12:08:53.480 INFO:teuthology.orchestra.run.smithi079.stdout:Waiting for the mgr to restart... 2024-05-04T12:08:53.480 INFO:teuthology.orchestra.run.smithi079.stdout:Waiting for mgr epoch 9... 2024-05-04T12:08:53.899 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:08:53 smithi079 ceph-mon[27222]: from='client.? 172.21.15.79:0/568999833' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-05-04T12:08:58.695 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:08:58 smithi079 ceph-mon[27222]: Active manager daemon smithi079.dgcpwl restarted 2024-05-04T12:08:58.695 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:08:58 smithi079 ceph-mon[27222]: Activating manager daemon smithi079.dgcpwl 2024-05-04T12:08:58.695 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:08:58 smithi079 ceph-mon[27222]: osdmap e3: 0 total, 0 up, 0 in 2024-05-04T12:08:58.695 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:08:58 smithi079 ceph-mon[27222]: mgrmap e10: smithi079.dgcpwl(active, starting, since 0.00397474s) 2024-05-04T12:08:58.695 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:08:58 smithi079 ceph-mon[27222]: from='mgr.14164 172.21.15.79:0/395052435' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "mon metadata", "id": "smithi079"}]: dispatch 2024-05-04T12:08:58.695 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:08:58 smithi079 ceph-mon[27222]: from='mgr.14164 172.21.15.79:0/395052435' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "mgr metadata", "who": "smithi079.dgcpwl", "id": "smithi079.dgcpwl"}]: dispatch 2024-05-04T12:08:58.696 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:08:58 smithi079 ceph-mon[27222]: from='mgr.14164 172.21.15.79:0/395052435' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-04T12:08:58.696 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:08:58 smithi079 ceph-mon[27222]: from='mgr.14164 172.21.15.79:0/395052435' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-04T12:08:58.696 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:08:58 smithi079 ceph-mon[27222]: from='mgr.14164 172.21.15.79:0/395052435' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-04T12:08:58.696 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:08:58 smithi079 ceph-mon[27222]: Manager daemon smithi079.dgcpwl is now available 2024-05-04T12:08:58.696 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:08:58 smithi079 ceph-mon[27222]: from='mgr.14164 172.21.15.79:0/395052435' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-04T12:08:58.696 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:08:58 smithi079 ceph-mon[27222]: from='mgr.14164 172.21.15.79:0/395052435' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-04T12:08:58.696 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:08:58 smithi079 ceph-mon[27222]: from='mgr.14164 172.21.15.79:0/395052435' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi079.dgcpwl/mirror_snapshot_schedule"}]: dispatch 2024-05-04T12:08:58.696 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:08:58 smithi079 ceph-mon[27222]: from='mgr.14164 172.21.15.79:0/395052435' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi079.dgcpwl/trash_purge_schedule"}]: dispatch 2024-05-04T12:08:59.844 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout { 2024-05-04T12:08:59.844 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-05-04T12:08:59.845 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "initialized": true 2024-05-04T12:08:59.845 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout } 2024-05-04T12:08:59.845 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:54.113+0000 7f1fd907d700 1 Processor -- start 2024-05-04T12:08:59.845 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:54.113+0000 7f1fd907d700 1 -- start start 2024-05-04T12:08:59.845 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:54.113+0000 7f1fd907d700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1fd4106560 0x7f1fd4106980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:08:59.845 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:54.113+0000 7f1fd907d700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fd4106f50 con 0x7f1fd4106560 2024-05-04T12:08:59.845 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:54.113+0000 7f1fd2d9d700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1fd4106560 0x7f1fd4106980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:08:59.845 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:54.113+0000 7f1fd2d9d700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1fd4106560 0x7f1fd4106980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:36300/0 (socket says 172.21.15.79:36300) 2024-05-04T12:08:59.845 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:54.113+0000 7f1fd2d9d700 1 -- 172.21.15.79:0/4145507138 learned_addr learned my addr 172.21.15.79:0/4145507138 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:08:59.845 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:54.113+0000 7f1fd2d9d700 1 -- 172.21.15.79:0/4145507138 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fd4107760 con 0x7f1fd4106560 2024-05-04T12:08:59.845 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:54.114+0000 7f1fd2d9d700 1 --2- 172.21.15.79:0/4145507138 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1fd4106560 0x7f1fd4106980 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f1fc40098a0 tx=0x7f1fc4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dba9bf09dbd6c8b3 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:08:59.845 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:54.114+0000 7f1fd1d9b700 1 -- 172.21.15.79:0/4145507138 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1fc4004180 con 0x7f1fd4106560 2024-05-04T12:08:59.845 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:54.114+0000 7f1fd1d9b700 1 -- 172.21.15.79:0/4145507138 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f1fc40042e0 con 0x7f1fd4106560 2024-05-04T12:08:59.845 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:54.114+0000 7f1fd1d9b700 1 -- 172.21.15.79:0/4145507138 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1fc4011af0 con 0x7f1fd4106560 2024-05-04T12:08:59.846 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:54.114+0000 7f1fd907d700 1 -- 172.21.15.79:0/4145507138 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1fd4106560 msgr2=0x7f1fd4106980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:08:59.846 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:54.114+0000 7f1fd907d700 1 --2- 172.21.15.79:0/4145507138 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1fd4106560 0x7f1fd4106980 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f1fc40098a0 tx=0x7f1fc4009bb0 comp rx=0 tx=0).stop 2024-05-04T12:08:59.846 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:54.114+0000 7f1fd907d700 1 -- 172.21.15.79:0/4145507138 shutdown_connections 2024-05-04T12:08:59.846 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:54.114+0000 7f1fd907d700 1 --2- 172.21.15.79:0/4145507138 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1fd4106560 0x7f1fd4106980 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:08:59.846 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:54.114+0000 7f1fd907d700 1 -- 172.21.15.79:0/4145507138 >> 172.21.15.79:0/4145507138 conn(0x7f1fd4101b20 msgr2=0x7f1fd4103f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:08:59.846 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:54.114+0000 7f1fd907d700 1 -- 172.21.15.79:0/4145507138 shutdown_connections 2024-05-04T12:08:59.846 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:54.114+0000 7f1fd907d700 1 -- 172.21.15.79:0/4145507138 wait complete. 2024-05-04T12:08:59.846 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:54.115+0000 7f1fd907d700 1 Processor -- start 2024-05-04T12:08:59.846 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:54.115+0000 7f1fd907d700 1 -- start start 2024-05-04T12:08:59.846 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:54.115+0000 7f1fd907d700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1fd4106560 0x7f1fd407ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:08:59.846 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:54.115+0000 7f1fd907d700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fd407b060 con 0x7f1fd4106560 2024-05-04T12:08:59.846 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:54.116+0000 7f1fd2d9d700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1fd4106560 0x7f1fd407ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:08:59.846 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:54.116+0000 7f1fd2d9d700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1fd4106560 0x7f1fd407ab20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:36306/0 (socket says 172.21.15.79:36306) 2024-05-04T12:08:59.846 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:54.116+0000 7f1fd2d9d700 1 -- 172.21.15.79:0/818326172 learned_addr learned my addr 172.21.15.79:0/818326172 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:08:59.847 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:54.116+0000 7f1fd2d9d700 1 -- 172.21.15.79:0/818326172 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fc4009550 con 0x7f1fd4106560 2024-05-04T12:08:59.847 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:54.116+0000 7f1fd2d9d700 1 --2- 172.21.15.79:0/818326172 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1fd4106560 0x7f1fd407ab20 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f1fc4000c00 tx=0x7f1fc4011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:08:59.847 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:54.116+0000 7f1fbbfff700 1 -- 172.21.15.79:0/818326172 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1fc4003a70 con 0x7f1fd4106560 2024-05-04T12:08:59.847 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:54.116+0000 7f1fbbfff700 1 -- 172.21.15.79:0/818326172 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f1fc4003bd0 con 0x7f1fd4106560 2024-05-04T12:08:59.847 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:54.116+0000 7f1fd907d700 1 -- 172.21.15.79:0/818326172 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fd4079170 con 0x7f1fd4106560 2024-05-04T12:08:59.847 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:54.116+0000 7f1fbbfff700 1 -- 172.21.15.79:0/818326172 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1fc40238c0 con 0x7f1fd4106560 2024-05-04T12:08:59.847 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:54.116+0000 7f1fd907d700 1 -- 172.21.15.79:0/818326172 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fd4079610 con 0x7f1fd4106560 2024-05-04T12:08:59.847 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:54.117+0000 7f1fbbfff700 1 -- 172.21.15.79:0/818326172 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 9) v1 ==== 45296+0+0 (secure 0 0 0) 0x7f1fc4023b80 con 0x7f1fd4106560 2024-05-04T12:08:59.847 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:54.117+0000 7f1fbbfff700 1 --2- 172.21.15.79:0/818326172 >> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] conn(0x7f1fbc038370 0x7f1fbc03a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:08:59.847 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:54.117+0000 7f1fd907d700 1 -- 172.21.15.79:0/818326172 --> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1fd40623c0 con 0x7f1fbc038370 2024-05-04T12:08:59.847 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:54.117+0000 7f1fbbfff700 1 -- 172.21.15.79:0/818326172 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f1fc404d1f0 con 0x7f1fd4106560 2024-05-04T12:08:59.847 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:54.117+0000 7f1fd259c700 1 -- 172.21.15.79:0/818326172 >> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] conn(0x7f1fbc038370 msgr2=0x7f1fbc03a830 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.79:6800/1620705054 2024-05-04T12:08:59.847 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:54.118+0000 7f1fd259c700 1 --2- 172.21.15.79:0/818326172 >> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] conn(0x7f1fbc038370 0x7f1fbc03a830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-04T12:08:59.847 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:54.319+0000 7f1fd259c700 1 -- 172.21.15.79:0/818326172 >> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] conn(0x7f1fbc038370 msgr2=0x7f1fbc03a830 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.79:6800/1620705054 2024-05-04T12:08:59.848 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:54.319+0000 7f1fd259c700 1 --2- 172.21.15.79:0/818326172 >> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] conn(0x7f1fbc038370 0x7f1fbc03a830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-04T12:08:59.848 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:54.720+0000 7f1fd259c700 1 -- 172.21.15.79:0/818326172 >> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] conn(0x7f1fbc038370 msgr2=0x7f1fbc03a830 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.79:6800/1620705054 2024-05-04T12:08:59.848 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:54.720+0000 7f1fd259c700 1 --2- 172.21.15.79:0/818326172 >> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] conn(0x7f1fbc038370 0x7f1fbc03a830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-04T12:08:59.848 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:55.521+0000 7f1fd259c700 1 -- 172.21.15.79:0/818326172 >> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] conn(0x7f1fbc038370 msgr2=0x7f1fbc03a830 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.79:6800/1620705054 2024-05-04T12:08:59.848 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:55.521+0000 7f1fd259c700 1 --2- 172.21.15.79:0/818326172 >> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] conn(0x7f1fbc038370 0x7f1fbc03a830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-04T12:08:59.848 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:57.122+0000 7f1fd259c700 1 -- 172.21.15.79:0/818326172 >> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] conn(0x7f1fbc038370 msgr2=0x7f1fbc03a830 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.79:6800/1620705054 2024-05-04T12:08:59.848 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:57.122+0000 7f1fd259c700 1 --2- 172.21.15.79:0/818326172 >> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] conn(0x7f1fbc038370 0x7f1fbc03a830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-05-04T12:08:59.848 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:58.219+0000 7f1fbbfff700 1 -- 172.21.15.79:0/818326172 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mgrmap(e 10) v1 ==== 45063+0+0 (secure 0 0 0) 0x7f1fc402ace0 con 0x7f1fd4106560 2024-05-04T12:08:59.848 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:58.219+0000 7f1fbbfff700 1 -- 172.21.15.79:0/818326172 >> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] conn(0x7f1fbc038370 msgr2=0x7f1fbc03a830 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-04T12:08:59.848 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:58.219+0000 7f1fbbfff700 1 --2- 172.21.15.79:0/818326172 >> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] conn(0x7f1fbc038370 0x7f1fbc03a830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:08:59.848 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:59.220+0000 7f1fbbfff700 1 -- 172.21.15.79:0/818326172 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mgrmap(e 11) v1 ==== 45190+0+0 (secure 0 0 0) 0x7f1fc4019c60 con 0x7f1fd4106560 2024-05-04T12:08:59.848 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:59.220+0000 7f1fbbfff700 1 --2- 172.21.15.79:0/818326172 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7f1fbc03bc30 0x7f1fbc03e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:08:59.848 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:59.220+0000 7f1fbbfff700 1 -- 172.21.15.79:0/818326172 --> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1fd40623c0 con 0x7f1fbc03bc30 2024-05-04T12:08:59.849 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:59.220+0000 7f1fd259c700 1 --2- 172.21.15.79:0/818326172 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7f1fbc03bc30 0x7f1fbc03e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:08:59.849 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:59.221+0000 7f1fd259c700 1 --2- 172.21.15.79:0/818326172 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7f1fbc03bc30 0x7f1fbc03e020 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f1fc8003a10 tx=0x7f1fc80092b0 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:08:59.849 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:59.222+0000 7f1fbbfff700 1 -- 172.21.15.79:0/818326172 <== mgr.14164 v2:172.21.15.79:6800/4263681324 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6910 (secure 0 0 0) 0x7f1fd40623c0 con 0x7f1fbc03bc30 2024-05-04T12:08:59.849 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:59.228+0000 7f1fd907d700 1 -- 172.21.15.79:0/818326172 --> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f1fd407a6c0 con 0x7f1fbc03bc30 2024-05-04T12:08:59.849 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:59.228+0000 7f1fbbfff700 1 -- 172.21.15.79:0/818326172 <== mgr.14164 v2:172.21.15.79:6800/4263681324 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7f1fd407a6c0 con 0x7f1fbc03bc30 2024-05-04T12:08:59.849 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:59.228+0000 7f1fd907d700 1 -- 172.21.15.79:0/818326172 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7f1fbc03bc30 msgr2=0x7f1fbc03e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:08:59.849 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:59.228+0000 7f1fd907d700 1 --2- 172.21.15.79:0/818326172 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7f1fbc03bc30 0x7f1fbc03e020 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f1fc8003a10 tx=0x7f1fc80092b0 comp rx=0 tx=0).stop 2024-05-04T12:08:59.849 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:59.228+0000 7f1fd907d700 1 -- 172.21.15.79:0/818326172 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1fd4106560 msgr2=0x7f1fd407ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:08:59.849 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:59.228+0000 7f1fd907d700 1 --2- 172.21.15.79:0/818326172 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1fd4106560 0x7f1fd407ab20 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f1fc4000c00 tx=0x7f1fc4011b50 comp rx=0 tx=0).stop 2024-05-04T12:08:59.849 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:59.228+0000 7f1fd907d700 1 -- 172.21.15.79:0/818326172 shutdown_connections 2024-05-04T12:08:59.849 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:59.228+0000 7f1fd907d700 1 --2- 172.21.15.79:0/818326172 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7f1fbc03bc30 0x7f1fbc03e020 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:08:59.849 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:59.228+0000 7f1fd907d700 1 --2- 172.21.15.79:0/818326172 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1fd4106560 0x7f1fd407ab20 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:08:59.849 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:59.228+0000 7f1fd907d700 1 --2- 172.21.15.79:0/818326172 >> [v2:172.21.15.79:6800/1620705054,v1:172.21.15.79:6801/1620705054] conn(0x7f1fbc038370 0x7f1fbc03a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:08:59.849 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:59.228+0000 7f1fd907d700 1 -- 172.21.15.79:0/818326172 >> 172.21.15.79:0/818326172 conn(0x7f1fd4101b20 msgr2=0x7f1fd4102860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:08:59.849 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:59.228+0000 7f1fd907d700 1 -- 172.21.15.79:0/818326172 shutdown_connections 2024-05-04T12:08:59.850 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:08:59.228+0000 7f1fd907d700 1 -- 172.21.15.79:0/818326172 wait complete. 2024-05-04T12:08:59.850 INFO:teuthology.orchestra.run.smithi079.stdout:mgr epoch 9 is available 2024-05-04T12:08:59.850 INFO:teuthology.orchestra.run.smithi079.stdout:Generating a dashboard self-signed certificate... 2024-05-04T12:09:00.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:09:00 smithi079 ceph-mon[27222]: [04/May/2024:12:08:58] ENGINE Bus STARTING 2024-05-04T12:09:00.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:09:00 smithi079 ceph-mon[27222]: [04/May/2024:12:08:58] ENGINE Serving on http://172.21.15.79:8765 2024-05-04T12:09:00.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:09:00 smithi079 ceph-mon[27222]: [04/May/2024:12:08:58] ENGINE Serving on https://172.21.15.79:7150 2024-05-04T12:09:00.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:09:00 smithi079 ceph-mon[27222]: [04/May/2024:12:08:58] ENGINE Bus STARTED 2024-05-04T12:09:00.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:09:00 smithi079 ceph-mon[27222]: from='client.14168 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-05-04T12:09:00.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:09:00 smithi079 ceph-mon[27222]: mgrmap e11: smithi079.dgcpwl(active, since 1.00551s) 2024-05-04T12:09:00.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:09:00 smithi079 ceph-mon[27222]: from='client.14168 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-05-04T12:09:01.165 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-05-04T12:09:01.165 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:00.239+0000 7f17eb5e4700 1 Processor -- start 2024-05-04T12:09:01.165 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:00.240+0000 7f17eb5e4700 1 -- start start 2024-05-04T12:09:01.166 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:00.240+0000 7f17eb5e4700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f17e4106560 0x7f17e4106980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:09:01.166 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:00.240+0000 7f17eb5e4700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17e4106f50 con 0x7f17e4106560 2024-05-04T12:09:01.166 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:00.240+0000 7f17e9380700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f17e4106560 0x7f17e4106980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:09:01.166 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:00.240+0000 7f17e9380700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f17e4106560 0x7f17e4106980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:36370/0 (socket says 172.21.15.79:36370) 2024-05-04T12:09:01.166 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:00.240+0000 7f17e9380700 1 -- 172.21.15.79:0/1953737591 learned_addr learned my addr 172.21.15.79:0/1953737591 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:09:01.166 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:00.240+0000 7f17e9380700 1 -- 172.21.15.79:0/1953737591 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17e4107760 con 0x7f17e4106560 2024-05-04T12:09:01.166 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:00.240+0000 7f17e9380700 1 --2- 172.21.15.79:0/1953737591 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f17e4106560 0x7f17e4106980 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f17dc009e80 tx=0x7f17dc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77e0057cc9925057 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:09:01.166 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:00.241+0000 7f17e3fff700 1 -- 172.21.15.79:0/1953737591 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f17dc00e070 con 0x7f17e4106560 2024-05-04T12:09:01.166 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:00.241+0000 7f17e3fff700 1 -- 172.21.15.79:0/1953737591 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f17dc00b8c0 con 0x7f17e4106560 2024-05-04T12:09:01.166 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:00.241+0000 7f17e3fff700 1 -- 172.21.15.79:0/1953737591 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f17dc0044d0 con 0x7f17e4106560 2024-05-04T12:09:01.166 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:00.241+0000 7f17eb5e4700 1 -- 172.21.15.79:0/1953737591 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f17e4106560 msgr2=0x7f17e4106980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:09:01.166 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:00.241+0000 7f17eb5e4700 1 --2- 172.21.15.79:0/1953737591 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f17e4106560 0x7f17e4106980 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f17dc009e80 tx=0x7f17dc00b330 comp rx=0 tx=0).stop 2024-05-04T12:09:01.167 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:00.241+0000 7f17eb5e4700 1 -- 172.21.15.79:0/1953737591 shutdown_connections 2024-05-04T12:09:01.167 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:00.241+0000 7f17eb5e4700 1 --2- 172.21.15.79:0/1953737591 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f17e4106560 0x7f17e4106980 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:09:01.167 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:00.241+0000 7f17eb5e4700 1 -- 172.21.15.79:0/1953737591 >> 172.21.15.79:0/1953737591 conn(0x7f17e4101b40 msgr2=0x7f17e4103f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:09:01.167 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:00.241+0000 7f17eb5e4700 1 -- 172.21.15.79:0/1953737591 shutdown_connections 2024-05-04T12:09:01.167 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:00.241+0000 7f17eb5e4700 1 -- 172.21.15.79:0/1953737591 wait complete. 2024-05-04T12:09:01.167 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:00.242+0000 7f17eb5e4700 1 Processor -- start 2024-05-04T12:09:01.167 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:00.242+0000 7f17eb5e4700 1 -- start start 2024-05-04T12:09:01.167 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:00.242+0000 7f17eb5e4700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f17e4106560 0x7f17e419a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:09:01.167 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:00.242+0000 7f17eb5e4700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17e419a740 con 0x7f17e4106560 2024-05-04T12:09:01.167 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:00.242+0000 7f17e9380700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f17e4106560 0x7f17e419a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:09:01.168 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:00.242+0000 7f17e9380700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f17e4106560 0x7f17e419a200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:36386/0 (socket says 172.21.15.79:36386) 2024-05-04T12:09:01.168 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:00.242+0000 7f17e9380700 1 -- 172.21.15.79:0/807560869 learned_addr learned my addr 172.21.15.79:0/807560869 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:09:01.168 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:00.242+0000 7f17e9380700 1 -- 172.21.15.79:0/807560869 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17dc009b30 con 0x7f17e4106560 2024-05-04T12:09:01.168 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:00.242+0000 7f17e9380700 1 --2- 172.21.15.79:0/807560869 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f17e4106560 0x7f17e419a200 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f17dc004940 tx=0x7f17dc004a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:09:01.168 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:00.243+0000 7f17e27fc700 1 -- 172.21.15.79:0/807560869 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f17dc00e040 con 0x7f17e4106560 2024-05-04T12:09:01.168 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:00.243+0000 7f17eb5e4700 1 -- 172.21.15.79:0/807560869 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17e419a940 con 0x7f17e4106560 2024-05-04T12:09:01.168 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:00.243+0000 7f17e27fc700 1 -- 172.21.15.79:0/807560869 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f17dc00bc40 con 0x7f17e4106560 2024-05-04T12:09:01.168 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:00.243+0000 7f17e27fc700 1 -- 172.21.15.79:0/807560869 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f17dc012670 con 0x7f17e4106560 2024-05-04T12:09:01.168 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:00.243+0000 7f17eb5e4700 1 -- 172.21.15.79:0/807560869 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17e419ae40 con 0x7f17e4106560 2024-05-04T12:09:01.168 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:00.243+0000 7f17e27fc700 1 -- 172.21.15.79:0/807560869 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 12) v1 ==== 45296+0+0 (secure 0 0 0) 0x7f17dc019040 con 0x7f17e4106560 2024-05-04T12:09:01.168 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:00.243+0000 7f17eb5e4700 1 -- 172.21.15.79:0/807560869 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17e40623c0 con 0x7f17e4106560 2024-05-04T12:09:01.169 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:00.244+0000 7f17e27fc700 1 --2- 172.21.15.79:0/807560869 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7f17cc038370 0x7f17cc03a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:09:01.169 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:00.244+0000 7f17e27fc700 1 -- 172.21.15.79:0/807560869 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f17dc04be60 con 0x7f17e4106560 2024-05-04T12:09:01.169 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:00.244+0000 7f17e8b7f700 1 --2- 172.21.15.79:0/807560869 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7f17cc038370 0x7f17cc03a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:09:01.169 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:00.244+0000 7f17e8b7f700 1 --2- 172.21.15.79:0/807560869 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7f17cc038370 0x7f17cc03a830 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f17d8006fd0 tx=0x7f17d8006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:09:01.169 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:00.248+0000 7f17e27fc700 1 -- 172.21.15.79:0/807560869 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f17dc011750 con 0x7f17e4106560 2024-05-04T12:09:01.169 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:00.458+0000 7f17eb5e4700 1 -- 172.21.15.79:0/807560869 --> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7f17e4102fb0 con 0x7f17cc038370 2024-05-04T12:09:01.169 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:00.577+0000 7f17e27fc700 1 -- 172.21.15.79:0/807560869 <== mgr.14164 v2:172.21.15.79:6800/4263681324 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7f17e4102fb0 con 0x7f17cc038370 2024-05-04T12:09:01.169 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:00.579+0000 7f17eb5e4700 1 -- 172.21.15.79:0/807560869 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7f17cc038370 msgr2=0x7f17cc03a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:09:01.169 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:00.579+0000 7f17eb5e4700 1 --2- 172.21.15.79:0/807560869 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7f17cc038370 0x7f17cc03a830 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f17d8006fd0 tx=0x7f17d8006e40 comp rx=0 tx=0).stop 2024-05-04T12:09:01.169 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:00.579+0000 7f17eb5e4700 1 -- 172.21.15.79:0/807560869 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f17e4106560 msgr2=0x7f17e419a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:09:01.169 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:00.579+0000 7f17eb5e4700 1 --2- 172.21.15.79:0/807560869 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f17e4106560 0x7f17e419a200 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f17dc004940 tx=0x7f17dc004a20 comp rx=0 tx=0).stop 2024-05-04T12:09:01.169 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:00.579+0000 7f17eb5e4700 1 -- 172.21.15.79:0/807560869 shutdown_connections 2024-05-04T12:09:01.169 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:00.579+0000 7f17eb5e4700 1 --2- 172.21.15.79:0/807560869 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7f17cc038370 0x7f17cc03a830 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:09:01.170 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:00.579+0000 7f17eb5e4700 1 --2- 172.21.15.79:0/807560869 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f17e4106560 0x7f17e419a200 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:09:01.170 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:00.579+0000 7f17eb5e4700 1 -- 172.21.15.79:0/807560869 >> 172.21.15.79:0/807560869 conn(0x7f17e4101b40 msgr2=0x7f17e41028a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:09:01.170 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:00.579+0000 7f17eb5e4700 1 -- 172.21.15.79:0/807560869 shutdown_connections 2024-05-04T12:09:01.170 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:00.579+0000 7f17eb5e4700 1 -- 172.21.15.79:0/807560869 wait complete. 2024-05-04T12:09:01.170 INFO:teuthology.orchestra.run.smithi079.stdout:Creating initial admin user... 2024-05-04T12:09:01.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:09:01 smithi079 ceph-mon[27222]: mgrmap e12: smithi079.dgcpwl(active, since 2s) 2024-05-04T12:09:01.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:09:01 smithi079 ceph-mon[27222]: from='mgr.14164 172.21.15.79:0/395052435' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:09:01.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:09:01 smithi079 ceph-mon[27222]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-05-04T12:09:01.446 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:09:01 smithi079 ceph-mon[27222]: from='mgr.14164 172.21.15.79:0/395052435' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:09:01.446 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:09:01 smithi079 ceph-mon[27222]: from='mgr.14164 172.21.15.79:0/395052435' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:09:02.945 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:09:02 smithi079 ceph-mon[27222]: from='mgr.14164 172.21.15.79:0/395052435' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:09:02.945 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:09:02 smithi079 ceph-mon[27222]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-04T12:09:02.945 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:09:02 smithi079 ceph-mon[27222]: from='mgr.14164 172.21.15.79:0/395052435' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:09:03.563 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$oE/fNeyvjgQ4X5puYCzDl.WKA.yvW4ltRtkyWMyyroyK6IT1FcqQ2", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1714824542, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-05-04T12:09:03.563 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:01.576+0000 7f28b3fff700 1 Processor -- start 2024-05-04T12:09:03.563 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:01.576+0000 7f28b3fff700 1 -- start start 2024-05-04T12:09:03.563 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:01.577+0000 7f28b3fff700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f28b41068a0 0x7f28b4106cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:09:03.563 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:01.577+0000 7f28b3fff700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28b4107290 con 0x7f28b41068a0 2024-05-04T12:09:03.563 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:01.577+0000 7f28b2ffd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f28b41068a0 0x7f28b4106cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:09:03.564 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:01.577+0000 7f28b2ffd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f28b41068a0 0x7f28b4106cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:36390/0 (socket says 172.21.15.79:36390) 2024-05-04T12:09:03.564 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:01.577+0000 7f28b2ffd700 1 -- 172.21.15.79:0/1025546108 learned_addr learned my addr 172.21.15.79:0/1025546108 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:09:03.564 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:01.577+0000 7f28b2ffd700 1 -- 172.21.15.79:0/1025546108 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28b41054b0 con 0x7f28b41068a0 2024-05-04T12:09:03.564 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:01.577+0000 7f28b2ffd700 1 --2- 172.21.15.79:0/1025546108 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f28b41068a0 0x7f28b4106cc0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f28a40098a0 tx=0x7f28a4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5300c565636497d server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:09:03.564 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:01.577+0000 7f28b1ffb700 1 -- 172.21.15.79:0/1025546108 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f28a4004180 con 0x7f28b41068a0 2024-05-04T12:09:03.564 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:01.577+0000 7f28b1ffb700 1 -- 172.21.15.79:0/1025546108 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f28a40042e0 con 0x7f28b41068a0 2024-05-04T12:09:03.564 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:01.577+0000 7f28b1ffb700 1 -- 172.21.15.79:0/1025546108 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f28a4011af0 con 0x7f28b41068a0 2024-05-04T12:09:03.564 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:01.578+0000 7f28b3fff700 1 -- 172.21.15.79:0/1025546108 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f28b41068a0 msgr2=0x7f28b4106cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:09:03.564 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:01.578+0000 7f28b3fff700 1 --2- 172.21.15.79:0/1025546108 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f28b41068a0 0x7f28b4106cc0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f28a40098a0 tx=0x7f28a4009bb0 comp rx=0 tx=0).stop 2024-05-04T12:09:03.564 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:01.578+0000 7f28b3fff700 1 -- 172.21.15.79:0/1025546108 shutdown_connections 2024-05-04T12:09:03.564 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:01.578+0000 7f28b3fff700 1 --2- 172.21.15.79:0/1025546108 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f28b41068a0 0x7f28b4106cc0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:09:03.564 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:01.578+0000 7f28b3fff700 1 -- 172.21.15.79:0/1025546108 >> 172.21.15.79:0/1025546108 conn(0x7f28b40fffe0 msgr2=0x7f28b4102420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:09:03.564 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:01.578+0000 7f28b3fff700 1 -- 172.21.15.79:0/1025546108 shutdown_connections 2024-05-04T12:09:03.564 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:01.578+0000 7f28b3fff700 1 -- 172.21.15.79:0/1025546108 wait complete. 2024-05-04T12:09:03.565 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:01.579+0000 7f28b3fff700 1 Processor -- start 2024-05-04T12:09:03.565 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:01.579+0000 7f28b3fff700 1 -- start start 2024-05-04T12:09:03.565 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:01.579+0000 7f28b3fff700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f28b41068a0 0x7f28b4104f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:09:03.565 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:01.579+0000 7f28b3fff700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28b419ce10 con 0x7f28b41068a0 2024-05-04T12:09:03.565 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:01.579+0000 7f28b2ffd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f28b41068a0 0x7f28b4104f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:09:03.565 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:01.579+0000 7f28b2ffd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f28b41068a0 0x7f28b4104f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:36406/0 (socket says 172.21.15.79:36406) 2024-05-04T12:09:03.565 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:01.579+0000 7f28b2ffd700 1 -- 172.21.15.79:0/1825343288 learned_addr learned my addr 172.21.15.79:0/1825343288 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:09:03.565 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:01.579+0000 7f28b2ffd700 1 -- 172.21.15.79:0/1825343288 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28a4009550 con 0x7f28b41068a0 2024-05-04T12:09:03.565 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:01.580+0000 7f28b2ffd700 1 --2- 172.21.15.79:0/1825343288 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f28b41068a0 0x7f28b4104f00 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f28a4011670 tx=0x7f28a4011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:09:03.565 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:01.580+0000 7f289bfff700 1 -- 172.21.15.79:0/1825343288 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f28a40039c0 con 0x7f28b41068a0 2024-05-04T12:09:03.565 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:01.580+0000 7f289bfff700 1 -- 172.21.15.79:0/1825343288 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f28a4003b20 con 0x7f28b41068a0 2024-05-04T12:09:03.565 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:01.580+0000 7f28b3fff700 1 -- 172.21.15.79:0/1825343288 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28b419cf50 con 0x7f28b41068a0 2024-05-04T12:09:03.565 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:01.580+0000 7f289bfff700 1 -- 172.21.15.79:0/1825343288 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f28a40247f0 con 0x7f28b41068a0 2024-05-04T12:09:03.566 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:01.580+0000 7f28b3fff700 1 -- 172.21.15.79:0/1825343288 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28b419d390 con 0x7f28b41068a0 2024-05-04T12:09:03.566 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:01.581+0000 7f289bfff700 1 -- 172.21.15.79:0/1825343288 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 12) v1 ==== 45296+0+0 (secure 0 0 0) 0x7f28a4024a60 con 0x7f28b41068a0 2024-05-04T12:09:03.566 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:01.581+0000 7f28b3fff700 1 -- 172.21.15.79:0/1825343288 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28b40623c0 con 0x7f28b41068a0 2024-05-04T12:09:03.566 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:01.581+0000 7f289bfff700 1 --2- 172.21.15.79:0/1825343288 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7f289c038380 0x7f289c03a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:09:03.566 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:01.581+0000 7f289bfff700 1 -- 172.21.15.79:0/1825343288 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f28a404c320 con 0x7f28b41068a0 2024-05-04T12:09:03.566 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:01.581+0000 7f28b27fc700 1 --2- 172.21.15.79:0/1825343288 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7f289c038380 0x7f289c03a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:09:03.566 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:01.582+0000 7f28b27fc700 1 --2- 172.21.15.79:0/1825343288 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7f289c038380 0x7f289c03a840 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f28a8006fd0 tx=0x7f28a8006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:09:03.566 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:01.586+0000 7f289bfff700 1 -- 172.21.15.79:0/1825343288 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f28a4022310 con 0x7f28b41068a0 2024-05-04T12:09:03.566 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:01.808+0000 7f28b3fff700 1 -- 172.21.15.79:0/1825343288 --> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f28b41096f0 con 0x7f289c038380 2024-05-04T12:09:03.566 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:02.037+0000 7f289bfff700 1 -- 172.21.15.79:0/1825343288 <== mgr.14164 v2:172.21.15.79:6800/4263681324 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f28b41096f0 con 0x7f289c038380 2024-05-04T12:09:03.566 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:02.040+0000 7f28b3fff700 1 -- 172.21.15.79:0/1825343288 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7f289c038380 msgr2=0x7f289c03a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:09:03.566 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:02.040+0000 7f28b3fff700 1 --2- 172.21.15.79:0/1825343288 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7f289c038380 0x7f289c03a840 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f28a8006fd0 tx=0x7f28a8006e40 comp rx=0 tx=0).stop 2024-05-04T12:09:03.566 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:02.040+0000 7f28b3fff700 1 -- 172.21.15.79:0/1825343288 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f28b41068a0 msgr2=0x7f28b4104f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:09:03.566 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:02.040+0000 7f28b3fff700 1 --2- 172.21.15.79:0/1825343288 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f28b41068a0 0x7f28b4104f00 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f28a4011670 tx=0x7f28a4011aa0 comp rx=0 tx=0).stop 2024-05-04T12:09:03.567 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:02.040+0000 7f28b3fff700 1 -- 172.21.15.79:0/1825343288 shutdown_connections 2024-05-04T12:09:03.567 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:02.040+0000 7f28b3fff700 1 --2- 172.21.15.79:0/1825343288 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7f289c038380 0x7f289c03a840 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:09:03.567 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:02.040+0000 7f28b3fff700 1 --2- 172.21.15.79:0/1825343288 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f28b41068a0 0x7f28b4104f00 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:09:03.567 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:02.040+0000 7f28b3fff700 1 -- 172.21.15.79:0/1825343288 >> 172.21.15.79:0/1825343288 conn(0x7f28b40fffe0 msgr2=0x7f28b4101660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:09:03.567 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:02.040+0000 7f28b3fff700 1 -- 172.21.15.79:0/1825343288 shutdown_connections 2024-05-04T12:09:03.567 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:02.040+0000 7f28b3fff700 1 -- 172.21.15.79:0/1825343288 wait complete. 2024-05-04T12:09:03.567 INFO:teuthology.orchestra.run.smithi079.stdout:Fetching dashboard port number... 2024-05-04T12:09:04.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:09:04 smithi079 ceph-mon[27222]: from='client.? 172.21.15.79:0/869554073' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-05-04T12:09:04.719 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout 8443 2024-05-04T12:09:04.719 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:03.992+0000 7f23b2512700 1 Processor -- start 2024-05-04T12:09:04.719 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:03.992+0000 7f23b2512700 1 -- start start 2024-05-04T12:09:04.719 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:03.992+0000 7f23b2512700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f23ac07be40 0x7f23ac07a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:09:04.719 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:03.992+0000 7f23b2512700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23ac07a870 con 0x7f23ac07be40 2024-05-04T12:09:04.719 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:03.993+0000 7f23abfff700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f23ac07be40 0x7f23ac07a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:09:04.719 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:03.993+0000 7f23abfff700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f23ac07be40 0x7f23ac07a2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:55428/0 (socket says 172.21.15.79:55428) 2024-05-04T12:09:04.719 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:03.993+0000 7f23abfff700 1 -- 172.21.15.79:0/1431481136 learned_addr learned my addr 172.21.15.79:0/1431481136 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:09:04.719 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:03.993+0000 7f23abfff700 1 -- 172.21.15.79:0/1431481136 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23ac07a9b0 con 0x7f23ac07be40 2024-05-04T12:09:04.720 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:03.993+0000 7f23abfff700 1 --2- 172.21.15.79:0/1431481136 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f23ac07be40 0x7f23ac07a2a0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f239c0098a0 tx=0x7f239c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4cc4fbeafe14564 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:09:04.720 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:03.993+0000 7f23aaffd700 1 -- 172.21.15.79:0/1431481136 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f239c004180 con 0x7f23ac07be40 2024-05-04T12:09:04.720 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:03.993+0000 7f23aaffd700 1 -- 172.21.15.79:0/1431481136 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f239c0042e0 con 0x7f23ac07be40 2024-05-04T12:09:04.720 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:03.993+0000 7f23aaffd700 1 -- 172.21.15.79:0/1431481136 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f239c011af0 con 0x7f23ac07be40 2024-05-04T12:09:04.720 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:03.994+0000 7f23b2512700 1 -- 172.21.15.79:0/1431481136 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f23ac07be40 msgr2=0x7f23ac07a2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:09:04.720 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:03.994+0000 7f23b2512700 1 --2- 172.21.15.79:0/1431481136 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f23ac07be40 0x7f23ac07a2a0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f239c0098a0 tx=0x7f239c009bb0 comp rx=0 tx=0).stop 2024-05-04T12:09:04.720 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:03.994+0000 7f23b2512700 1 -- 172.21.15.79:0/1431481136 shutdown_connections 2024-05-04T12:09:04.720 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:03.994+0000 7f23b2512700 1 --2- 172.21.15.79:0/1431481136 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f23ac07be40 0x7f23ac07a2a0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:09:04.720 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:03.994+0000 7f23b2512700 1 -- 172.21.15.79:0/1431481136 >> 172.21.15.79:0/1431481136 conn(0x7f23ac101b40 msgr2=0x7f23ac103f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:09:04.720 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:03.994+0000 7f23b2512700 1 -- 172.21.15.79:0/1431481136 shutdown_connections 2024-05-04T12:09:04.720 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:03.994+0000 7f23b2512700 1 -- 172.21.15.79:0/1431481136 wait complete. 2024-05-04T12:09:04.720 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:03.995+0000 7f23b2512700 1 Processor -- start 2024-05-04T12:09:04.720 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:03.995+0000 7f23b2512700 1 -- start start 2024-05-04T12:09:04.721 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:03.995+0000 7f23b2512700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f23ac07be40 0x7f23ac19c400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:09:04.721 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:03.995+0000 7f23b2512700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23ac19c940 con 0x7f23ac07be40 2024-05-04T12:09:04.721 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:03.995+0000 7f23abfff700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f23ac07be40 0x7f23ac19c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:09:04.721 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:03.995+0000 7f23abfff700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f23ac07be40 0x7f23ac19c400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:55434/0 (socket says 172.21.15.79:55434) 2024-05-04T12:09:04.721 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:03.995+0000 7f23abfff700 1 -- 172.21.15.79:0/869554073 learned_addr learned my addr 172.21.15.79:0/869554073 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:09:04.721 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:03.995+0000 7f23abfff700 1 -- 172.21.15.79:0/869554073 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f239c009550 con 0x7f23ac07be40 2024-05-04T12:09:04.721 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:03.995+0000 7f23abfff700 1 --2- 172.21.15.79:0/869554073 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f23ac07be40 0x7f23ac19c400 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f239c011670 tx=0x7f239c011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:09:04.721 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:03.996+0000 7f23a97fa700 1 -- 172.21.15.79:0/869554073 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f239c003a40 con 0x7f23ac07be40 2024-05-04T12:09:04.721 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:03.996+0000 7f23a97fa700 1 -- 172.21.15.79:0/869554073 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f239c003ba0 con 0x7f23ac07be40 2024-05-04T12:09:04.721 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:03.996+0000 7f23a97fa700 1 -- 172.21.15.79:0/869554073 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f239c024790 con 0x7f23ac07be40 2024-05-04T12:09:04.721 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:03.996+0000 7f23b2512700 1 -- 172.21.15.79:0/869554073 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23ac19cb40 con 0x7f23ac07be40 2024-05-04T12:09:04.721 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:03.996+0000 7f23b2512700 1 -- 172.21.15.79:0/869554073 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23ac19cfe0 con 0x7f23ac07be40 2024-05-04T12:09:04.721 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:03.997+0000 7f23a97fa700 1 -- 172.21.15.79:0/869554073 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 12) v1 ==== 45296+0+0 (secure 0 0 0) 0x7f239c02d3f0 con 0x7f23ac07be40 2024-05-04T12:09:04.721 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:03.997+0000 7f23b2512700 1 -- 172.21.15.79:0/869554073 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23ac10c900 con 0x7f23ac07be40 2024-05-04T12:09:04.722 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:03.997+0000 7f23a97fa700 1 --2- 172.21.15.79:0/869554073 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7f2394038370 0x7f239403a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:09:04.722 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:03.997+0000 7f23a97fa700 1 -- 172.21.15.79:0/869554073 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f239c04c5f0 con 0x7f23ac07be40 2024-05-04T12:09:04.722 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:03.997+0000 7f23ab7fe700 1 --2- 172.21.15.79:0/869554073 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7f2394038370 0x7f239403a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:09:04.722 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:03.997+0000 7f23ab7fe700 1 --2- 172.21.15.79:0/869554073 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7f2394038370 0x7f239403a830 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f23a0006fd0 tx=0x7f23a0006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:09:04.722 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:04.002+0000 7f23a97fa700 1 -- 172.21.15.79:0/869554073 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f239c018b40 con 0x7f23ac07be40 2024-05-04T12:09:04.722 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:04.190+0000 7f23b2512700 1 -- 172.21.15.79:0/869554073 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7f23ac10cb20 con 0x7f23ac07be40 2024-05-04T12:09:04.722 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:04.191+0000 7f23a97fa700 1 -- 172.21.15.79:0/869554073 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7f239c04be80 con 0x7f23ac07be40 2024-05-04T12:09:04.722 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:04.193+0000 7f23b2512700 1 -- 172.21.15.79:0/869554073 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7f2394038370 msgr2=0x7f239403a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:09:04.722 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:04.193+0000 7f23b2512700 1 --2- 172.21.15.79:0/869554073 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7f2394038370 0x7f239403a830 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f23a0006fd0 tx=0x7f23a0006e40 comp rx=0 tx=0).stop 2024-05-04T12:09:04.722 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:04.193+0000 7f23b2512700 1 -- 172.21.15.79:0/869554073 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f23ac07be40 msgr2=0x7f23ac19c400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:09:04.722 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:04.193+0000 7f23b2512700 1 --2- 172.21.15.79:0/869554073 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f23ac07be40 0x7f23ac19c400 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f239c011670 tx=0x7f239c011b50 comp rx=0 tx=0).stop 2024-05-04T12:09:04.722 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:04.193+0000 7f23b2512700 1 -- 172.21.15.79:0/869554073 shutdown_connections 2024-05-04T12:09:04.722 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:04.193+0000 7f23b2512700 1 --2- 172.21.15.79:0/869554073 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7f2394038370 0x7f239403a830 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:09:04.723 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:04.193+0000 7f23b2512700 1 --2- 172.21.15.79:0/869554073 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f23ac07be40 0x7f23ac19c400 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:09:04.723 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:04.193+0000 7f23b2512700 1 -- 172.21.15.79:0/869554073 >> 172.21.15.79:0/869554073 conn(0x7f23ac101b40 msgr2=0x7f23ac1028c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:09:04.723 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:04.193+0000 7f23b2512700 1 -- 172.21.15.79:0/869554073 shutdown_connections 2024-05-04T12:09:04.723 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:04.193+0000 7f23b2512700 1 -- 172.21.15.79:0/869554073 wait complete. 2024-05-04T12:09:04.726 INFO:teuthology.orchestra.run.smithi079.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-05-04T12:09:04.726 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stdout disabled 2024-05-04T12:09:04.733 INFO:teuthology.orchestra.run.smithi079.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-05-04T12:09:04.733 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stdout inactive 2024-05-04T12:09:04.733 INFO:teuthology.orchestra.run.smithi079.stdout:firewalld.service is not enabled 2024-05-04T12:09:04.733 INFO:teuthology.orchestra.run.smithi079.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-05-04T12:09:04.735 INFO:teuthology.orchestra.run.smithi079.stdout:Ceph Dashboard is now available at: 2024-05-04T12:09:04.735 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-05-04T12:09:04.735 INFO:teuthology.orchestra.run.smithi079.stdout: URL: https://smithi079:8443/ 2024-05-04T12:09:04.735 INFO:teuthology.orchestra.run.smithi079.stdout: User: admin 2024-05-04T12:09:04.735 INFO:teuthology.orchestra.run.smithi079.stdout: Password: msube0x6kd 2024-05-04T12:09:04.735 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-05-04T12:09:04.735 INFO:teuthology.orchestra.run.smithi079.stdout:Saving cluster configuration to /var/lib/ceph/d9b985c4-0a0e-11ef-bc97-c7b262605968/config directory 2024-05-04T12:09:04.736 INFO:teuthology.orchestra.run.smithi079.stdout:Enabling autotune for osd_memory_target 2024-05-04T12:09:05.444 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:09:05 smithi079 ceph-mon[27222]: mgrmap e13: smithi079.dgcpwl(active, since 6s) 2024-05-04T12:09:05.968 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:05.157+0000 7ff209979700 1 Processor -- start 2024-05-04T12:09:05.968 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:05.157+0000 7ff209979700 1 -- start start 2024-05-04T12:09:05.969 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:05.157+0000 7ff209979700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff204106580 0x7ff2041069a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:09:05.969 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:05.157+0000 7ff209979700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff204106f70 con 0x7ff204106580 2024-05-04T12:09:05.969 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:05.157+0000 7ff202ffd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff204106580 0x7ff2041069a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:09:05.969 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:05.157+0000 7ff202ffd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff204106580 0x7ff2041069a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:55442/0 (socket says 172.21.15.79:55442) 2024-05-04T12:09:05.969 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:05.157+0000 7ff202ffd700 1 -- 172.21.15.79:0/1615183519 learned_addr learned my addr 172.21.15.79:0/1615183519 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:09:05.969 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:05.157+0000 7ff202ffd700 1 -- 172.21.15.79:0/1615183519 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff204107780 con 0x7ff204106580 2024-05-04T12:09:05.969 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:05.157+0000 7ff202ffd700 1 --2- 172.21.15.79:0/1615183519 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff204106580 0x7ff2041069a0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7ff1f40098a0 tx=0x7ff1f4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5653638d200d4ce7 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:09:05.969 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:05.158+0000 7ff201ffb700 1 -- 172.21.15.79:0/1615183519 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff1f4004180 con 0x7ff204106580 2024-05-04T12:09:05.969 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:05.158+0000 7ff201ffb700 1 -- 172.21.15.79:0/1615183519 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7ff1f40042e0 con 0x7ff204106580 2024-05-04T12:09:05.969 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:05.158+0000 7ff201ffb700 1 -- 172.21.15.79:0/1615183519 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff1f4011af0 con 0x7ff204106580 2024-05-04T12:09:05.970 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:05.158+0000 7ff209979700 1 -- 172.21.15.79:0/1615183519 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff204106580 msgr2=0x7ff2041069a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:09:05.970 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:05.158+0000 7ff209979700 1 --2- 172.21.15.79:0/1615183519 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff204106580 0x7ff2041069a0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7ff1f40098a0 tx=0x7ff1f4009bb0 comp rx=0 tx=0).stop 2024-05-04T12:09:05.970 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:05.158+0000 7ff209979700 1 -- 172.21.15.79:0/1615183519 shutdown_connections 2024-05-04T12:09:05.970 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:05.158+0000 7ff209979700 1 --2- 172.21.15.79:0/1615183519 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff204106580 0x7ff2041069a0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:09:05.970 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:05.158+0000 7ff209979700 1 -- 172.21.15.79:0/1615183519 >> 172.21.15.79:0/1615183519 conn(0x7ff204101b00 msgr2=0x7ff204103f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:09:05.970 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:05.158+0000 7ff209979700 1 -- 172.21.15.79:0/1615183519 shutdown_connections 2024-05-04T12:09:05.970 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:05.158+0000 7ff209979700 1 -- 172.21.15.79:0/1615183519 wait complete. 2024-05-04T12:09:05.970 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:05.159+0000 7ff209979700 1 Processor -- start 2024-05-04T12:09:05.970 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:05.159+0000 7ff209979700 1 -- start start 2024-05-04T12:09:05.970 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:05.159+0000 7ff209979700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff204106580 0x7ff20419a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:09:05.970 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:05.159+0000 7ff209979700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff20419a7d0 con 0x7ff204106580 2024-05-04T12:09:05.970 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:05.159+0000 7ff202ffd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff204106580 0x7ff20419a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:09:05.971 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:05.159+0000 7ff202ffd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff204106580 0x7ff20419a290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:55450/0 (socket says 172.21.15.79:55450) 2024-05-04T12:09:05.971 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:05.159+0000 7ff202ffd700 1 -- 172.21.15.79:0/2171721455 learned_addr learned my addr 172.21.15.79:0/2171721455 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:09:05.971 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:05.160+0000 7ff202ffd700 1 -- 172.21.15.79:0/2171721455 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1f4009550 con 0x7ff204106580 2024-05-04T12:09:05.971 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:05.160+0000 7ff202ffd700 1 --2- 172.21.15.79:0/2171721455 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff204106580 0x7ff20419a290 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7ff1f4011670 tx=0x7ff1f4011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:09:05.971 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:05.160+0000 7ff208977700 1 -- 172.21.15.79:0/2171721455 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff1f40039c0 con 0x7ff204106580 2024-05-04T12:09:05.971 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:05.160+0000 7ff208977700 1 -- 172.21.15.79:0/2171721455 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7ff1f4003b20 con 0x7ff204106580 2024-05-04T12:09:05.971 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:05.160+0000 7ff208977700 1 -- 172.21.15.79:0/2171721455 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff1f40247f0 con 0x7ff204106580 2024-05-04T12:09:05.971 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:05.160+0000 7ff209979700 1 -- 172.21.15.79:0/2171721455 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff20419a9d0 con 0x7ff204106580 2024-05-04T12:09:05.971 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:05.160+0000 7ff209979700 1 -- 172.21.15.79:0/2171721455 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff20419ae70 con 0x7ff204106580 2024-05-04T12:09:05.972 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:05.161+0000 7ff208977700 1 -- 172.21.15.79:0/2171721455 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 13) v1 ==== 45339+0+0 (secure 0 0 0) 0x7ff1f401ac70 con 0x7ff204106580 2024-05-04T12:09:05.972 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:05.161+0000 7ff208977700 1 --2- 172.21.15.79:0/2171721455 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7ff1ec0383d0 0x7ff1ec03a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:09:05.972 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:05.161+0000 7ff209979700 1 -- 172.21.15.79:0/2171721455 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2040623c0 con 0x7ff204106580 2024-05-04T12:09:05.972 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:05.161+0000 7ff208977700 1 -- 172.21.15.79:0/2171721455 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7ff1f404c4d0 con 0x7ff204106580 2024-05-04T12:09:05.972 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:05.161+0000 7ff2027fc700 1 --2- 172.21.15.79:0/2171721455 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7ff1ec0383d0 0x7ff1ec03a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:09:05.972 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:05.161+0000 7ff2027fc700 1 --2- 172.21.15.79:0/2171721455 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7ff1ec0383d0 0x7ff1ec03a890 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7ff1f8006fd0 tx=0x7ff1f8006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:09:05.972 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:05.166+0000 7ff208977700 1 -- 172.21.15.79:0/2171721455 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7ff1f402d3f0 con 0x7ff204106580 2024-05-04T12:09:05.972 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:05.353+0000 7ff209979700 1 -- 172.21.15.79:0/2171721455 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7ff20410a980 con 0x7ff204106580 2024-05-04T12:09:05.972 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:05.353+0000 7ff208977700 1 -- 172.21.15.79:0/2171721455 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7ff1f403fd10 con 0x7ff204106580 2024-05-04T12:09:05.972 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:05.355+0000 7ff209979700 1 -- 172.21.15.79:0/2171721455 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7ff1ec0383d0 msgr2=0x7ff1ec03a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:09:05.972 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:05.355+0000 7ff209979700 1 --2- 172.21.15.79:0/2171721455 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7ff1ec0383d0 0x7ff1ec03a890 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7ff1f8006fd0 tx=0x7ff1f8006e40 comp rx=0 tx=0).stop 2024-05-04T12:09:05.973 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:05.355+0000 7ff209979700 1 -- 172.21.15.79:0/2171721455 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff204106580 msgr2=0x7ff20419a290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:09:05.973 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:05.355+0000 7ff209979700 1 --2- 172.21.15.79:0/2171721455 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff204106580 0x7ff20419a290 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7ff1f4011670 tx=0x7ff1f4011aa0 comp rx=0 tx=0).stop 2024-05-04T12:09:05.973 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:05.355+0000 7ff209979700 1 -- 172.21.15.79:0/2171721455 shutdown_connections 2024-05-04T12:09:05.973 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:05.355+0000 7ff209979700 1 --2- 172.21.15.79:0/2171721455 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7ff1ec0383d0 0x7ff1ec03a890 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:09:05.973 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:05.355+0000 7ff209979700 1 --2- 172.21.15.79:0/2171721455 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff204106580 0x7ff20419a290 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:09:05.973 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:05.355+0000 7ff209979700 1 -- 172.21.15.79:0/2171721455 >> 172.21.15.79:0/2171721455 conn(0x7ff204101b00 msgr2=0x7ff2041031e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:09:05.973 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:05.355+0000 7ff209979700 1 -- 172.21.15.79:0/2171721455 shutdown_connections 2024-05-04T12:09:05.973 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:05.355+0000 7ff209979700 1 -- 172.21.15.79:0/2171721455 wait complete. 2024-05-04T12:09:07.232 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:06.361+0000 7fe7ddf03700 1 Processor -- start 2024-05-04T12:09:07.232 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:06.361+0000 7fe7ddf03700 1 -- start start 2024-05-04T12:09:07.232 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:06.361+0000 7fe7ddf03700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe7d807bea0 0x7fe7d807a300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:09:07.232 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:06.361+0000 7fe7ddf03700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7d807a8d0 con 0x7fe7d807bea0 2024-05-04T12:09:07.232 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:06.361+0000 7fe7d77fe700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe7d807bea0 0x7fe7d807a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:09:07.232 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:06.361+0000 7fe7d77fe700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe7d807bea0 0x7fe7d807a300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:55452/0 (socket says 172.21.15.79:55452) 2024-05-04T12:09:07.233 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:06.361+0000 7fe7d77fe700 1 -- 172.21.15.79:0/2588035058 learned_addr learned my addr 172.21.15.79:0/2588035058 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:09:07.233 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:06.361+0000 7fe7d77fe700 1 -- 172.21.15.79:0/2588035058 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7d807aa10 con 0x7fe7d807bea0 2024-05-04T12:09:07.233 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:06.361+0000 7fe7d77fe700 1 --2- 172.21.15.79:0/2588035058 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe7d807bea0 0x7fe7d807a300 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fe7cc0098a0 tx=0x7fe7cc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ccb38dfc8d92a2f3 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:09:07.233 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:06.362+0000 7fe7d67fc700 1 -- 172.21.15.79:0/2588035058 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe7cc004180 con 0x7fe7d807bea0 2024-05-04T12:09:07.233 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:06.362+0000 7fe7d67fc700 1 -- 172.21.15.79:0/2588035058 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fe7cc0042e0 con 0x7fe7d807bea0 2024-05-04T12:09:07.233 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:06.362+0000 7fe7d67fc700 1 -- 172.21.15.79:0/2588035058 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe7cc011af0 con 0x7fe7d807bea0 2024-05-04T12:09:07.233 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:06.362+0000 7fe7ddf03700 1 -- 172.21.15.79:0/2588035058 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe7d807bea0 msgr2=0x7fe7d807a300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:09:07.233 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:06.362+0000 7fe7ddf03700 1 --2- 172.21.15.79:0/2588035058 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe7d807bea0 0x7fe7d807a300 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fe7cc0098a0 tx=0x7fe7cc009bb0 comp rx=0 tx=0).stop 2024-05-04T12:09:07.233 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:06.362+0000 7fe7ddf03700 1 -- 172.21.15.79:0/2588035058 shutdown_connections 2024-05-04T12:09:07.233 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:06.362+0000 7fe7ddf03700 1 --2- 172.21.15.79:0/2588035058 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe7d807bea0 0x7fe7d807a300 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:09:07.233 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:06.362+0000 7fe7ddf03700 1 -- 172.21.15.79:0/2588035058 >> 172.21.15.79:0/2588035058 conn(0x7fe7d8101b40 msgr2=0x7fe7d8103f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:09:07.233 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:06.362+0000 7fe7ddf03700 1 -- 172.21.15.79:0/2588035058 shutdown_connections 2024-05-04T12:09:07.233 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:06.362+0000 7fe7ddf03700 1 -- 172.21.15.79:0/2588035058 wait complete. 2024-05-04T12:09:07.233 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:06.363+0000 7fe7ddf03700 1 Processor -- start 2024-05-04T12:09:07.234 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:06.363+0000 7fe7ddf03700 1 -- start start 2024-05-04T12:09:07.234 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:06.363+0000 7fe7ddf03700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe7d807bea0 0x7fe7d819c480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:09:07.234 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:06.363+0000 7fe7ddf03700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7d819c9c0 con 0x7fe7d807bea0 2024-05-04T12:09:07.234 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:06.363+0000 7fe7d77fe700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe7d807bea0 0x7fe7d819c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:09:07.234 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:06.363+0000 7fe7d77fe700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe7d807bea0 0x7fe7d819c480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:55456/0 (socket says 172.21.15.79:55456) 2024-05-04T12:09:07.234 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:06.363+0000 7fe7d77fe700 1 -- 172.21.15.79:0/3929272773 learned_addr learned my addr 172.21.15.79:0/3929272773 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:09:07.234 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:06.364+0000 7fe7d77fe700 1 -- 172.21.15.79:0/3929272773 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7cc009550 con 0x7fe7d807bea0 2024-05-04T12:09:07.234 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:06.364+0000 7fe7d77fe700 1 --2- 172.21.15.79:0/3929272773 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe7d807bea0 0x7fe7d819c480 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fe7cc011670 tx=0x7fe7cc011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:09:07.234 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:06.364+0000 7fe7d4ff9700 1 -- 172.21.15.79:0/3929272773 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe7cc0039c0 con 0x7fe7d807bea0 2024-05-04T12:09:07.235 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:06.364+0000 7fe7d4ff9700 1 -- 172.21.15.79:0/3929272773 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fe7cc003b20 con 0x7fe7d807bea0 2024-05-04T12:09:07.235 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:06.364+0000 7fe7ddf03700 1 -- 172.21.15.79:0/3929272773 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7d819cbc0 con 0x7fe7d807bea0 2024-05-04T12:09:07.235 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:06.364+0000 7fe7d4ff9700 1 -- 172.21.15.79:0/3929272773 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe7cc0247f0 con 0x7fe7d807bea0 2024-05-04T12:09:07.235 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:06.364+0000 7fe7ddf03700 1 -- 172.21.15.79:0/3929272773 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7d819d060 con 0x7fe7d807bea0 2024-05-04T12:09:07.235 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:06.365+0000 7fe7d4ff9700 1 -- 172.21.15.79:0/3929272773 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 13) v1 ==== 45339+0+0 (secure 0 0 0) 0x7fe7cc024a60 con 0x7fe7d807bea0 2024-05-04T12:09:07.235 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:06.365+0000 7fe7d4ff9700 1 --2- 172.21.15.79:0/3929272773 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7fe7c0038420 0x7fe7c003a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:09:07.235 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:06.365+0000 7fe7d4ff9700 1 -- 172.21.15.79:0/3929272773 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fe7cc04c4d0 con 0x7fe7d807bea0 2024-05-04T12:09:07.235 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:06.365+0000 7fe7d6ffd700 1 --2- 172.21.15.79:0/3929272773 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7fe7c0038420 0x7fe7c003a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:09:07.235 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:06.365+0000 7fe7ddf03700 1 -- 172.21.15.79:0/3929272773 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7d80623c0 con 0x7fe7d807bea0 2024-05-04T12:09:07.235 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:06.365+0000 7fe7d6ffd700 1 --2- 172.21.15.79:0/3929272773 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7fe7c0038420 0x7fe7c003a8e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fe7c8006fd0 tx=0x7fe7c8006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:09:07.235 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:06.370+0000 7fe7d4ff9700 1 -- 172.21.15.79:0/3929272773 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fe7cc024d60 con 0x7fe7d807bea0 2024-05-04T12:09:07.235 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:06.667+0000 7fe7ddf03700 1 -- 172.21.15.79:0/3929272773 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7fe7d819fa20 con 0x7fe7d807bea0 2024-05-04T12:09:07.235 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:06.670+0000 7fe7d4ff9700 1 -- 172.21.15.79:0/3929272773 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v34)=0 set mgr/dashboard/cluster/status v34) v1 ==== 153+0+0 (secure 0 0 0) 0x7fe7cc020220 con 0x7fe7d807bea0 2024-05-04T12:09:07.236 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:06.672+0000 7fe7ddf03700 1 -- 172.21.15.79:0/3929272773 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7fe7c0038420 msgr2=0x7fe7c003a8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:09:07.236 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:06.672+0000 7fe7ddf03700 1 --2- 172.21.15.79:0/3929272773 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7fe7c0038420 0x7fe7c003a8e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fe7c8006fd0 tx=0x7fe7c8006e40 comp rx=0 tx=0).stop 2024-05-04T12:09:07.236 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:06.672+0000 7fe7ddf03700 1 -- 172.21.15.79:0/3929272773 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe7d807bea0 msgr2=0x7fe7d819c480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:09:07.236 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:06.672+0000 7fe7ddf03700 1 --2- 172.21.15.79:0/3929272773 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe7d807bea0 0x7fe7d819c480 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fe7cc011670 tx=0x7fe7cc011aa0 comp rx=0 tx=0).stop 2024-05-04T12:09:07.236 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:06.672+0000 7fe7ddf03700 1 -- 172.21.15.79:0/3929272773 shutdown_connections 2024-05-04T12:09:07.236 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:06.672+0000 7fe7ddf03700 1 --2- 172.21.15.79:0/3929272773 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7fe7c0038420 0x7fe7c003a8e0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:09:07.236 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:06.672+0000 7fe7ddf03700 1 --2- 172.21.15.79:0/3929272773 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe7d807bea0 0x7fe7d819c480 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:09:07.236 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:06.672+0000 7fe7ddf03700 1 -- 172.21.15.79:0/3929272773 >> 172.21.15.79:0/3929272773 conn(0x7fe7d8101b40 msgr2=0x7fe7d8103210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:09:07.236 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:06.672+0000 7fe7ddf03700 1 -- 172.21.15.79:0/3929272773 shutdown_connections 2024-05-04T12:09:07.236 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-05-04T12:09:06.672+0000 7fe7ddf03700 1 -- 172.21.15.79:0/3929272773 wait complete. 2024-05-04T12:09:07.236 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-05-04T12:09:07.236 INFO:teuthology.orchestra.run.smithi079.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-05-04T12:09:07.236 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-05-04T12:09:07.236 INFO:teuthology.orchestra.run.smithi079.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid d9b985c4-0a0e-11ef-bc97-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-05-04T12:09:07.237 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-05-04T12:09:07.237 INFO:teuthology.orchestra.run.smithi079.stdout:Or, if you are only running a single cluster on this host: 2024-05-04T12:09:07.237 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-05-04T12:09:07.237 INFO:teuthology.orchestra.run.smithi079.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-05-04T12:09:07.237 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-05-04T12:09:07.237 INFO:teuthology.orchestra.run.smithi079.stdout:Please consider enabling telemetry to help improve Ceph: 2024-05-04T12:09:07.237 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-05-04T12:09:07.237 INFO:teuthology.orchestra.run.smithi079.stdout: ceph telemetry on 2024-05-04T12:09:07.237 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-05-04T12:09:07.237 INFO:teuthology.orchestra.run.smithi079.stdout:For more information see: 2024-05-04T12:09:07.237 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-05-04T12:09:07.237 INFO:teuthology.orchestra.run.smithi079.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-05-04T12:09:07.237 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-05-04T12:09:07.237 INFO:teuthology.orchestra.run.smithi079.stdout:Bootstrap complete. 2024-05-04T12:09:07.262 INFO:tasks.cephadm:Fetching config... 2024-05-04T12:09:07.262 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-05-04T12:09:07.262 DEBUG:teuthology.orchestra.run.smithi079:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-05-04T12:09:07.277 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-05-04T12:09:07.277 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-05-04T12:09:07.277 DEBUG:teuthology.orchestra.run.smithi079:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-05-04T12:09:07.333 INFO:tasks.cephadm:Fetching mon keyring... 2024-05-04T12:09:07.333 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-05-04T12:09:07.333 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd if=/var/lib/ceph/d9b985c4-0a0e-11ef-bc97-c7b262605968/mon.smithi079/keyring of=/dev/stdout 2024-05-04T12:09:07.396 INFO:tasks.cephadm:Fetching pub ssh key... 2024-05-04T12:09:07.396 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-05-04T12:09:07.396 DEBUG:teuthology.orchestra.run.smithi079:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-05-04T12:09:07.450 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-05-04T12:09:07.450 DEBUG:teuthology.orchestra.run.smithi079:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCm7xcLdyPGUWO9LQ/mT1Bj8RF6DufxLVkza6YzpUq4GneHJDREQmAW/JWfepuD22rTR9AYtSiKp/G0PdhjHlVIf7A6NoWr3zuQFlBqe18A8sVglmLdRqex/E7LXYvPES34t8AmyLv+VOhRQ7gami3ArHe5RUV/C8XVjwksjitTyixV9Hl4V36A2vDWpkem49t0gMYY8OIOSRPBkq+P2KROawCZcONyfYApVbDL1548FIzbzfz3Lhlbok4PrOMs5TKkbw+/N+PT8NTB+wxpxN79gBYXoKQkt6MWUrKJLA2875Z6hXA1UF5MpY78P/fLjEtvMGCF0RRO7mv/I1EwGLao+AG/SkuVONFI8xE//aOKV4y/jm3xnimYSWawBXfnqeQNE+ZRs/T1NNP97jHmUWanqLDpbR6eDWXj2GY5heYL8w/r9ozypkns/G8yKpcy7fcGXwm0PI1scEikU+tw0G5G7nOxwGTqqMEhXgWNyah6q0z2rsaXTMxHyjxmVvxPYMU= ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-05-04T12:09:07.522 INFO:teuthology.orchestra.run.smithi079.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCm7xcLdyPGUWO9LQ/mT1Bj8RF6DufxLVkza6YzpUq4GneHJDREQmAW/JWfepuD22rTR9AYtSiKp/G0PdhjHlVIf7A6NoWr3zuQFlBqe18A8sVglmLdRqex/E7LXYvPES34t8AmyLv+VOhRQ7gami3ArHe5RUV/C8XVjwksjitTyixV9Hl4V36A2vDWpkem49t0gMYY8OIOSRPBkq+P2KROawCZcONyfYApVbDL1548FIzbzfz3Lhlbok4PrOMs5TKkbw+/N+PT8NTB+wxpxN79gBYXoKQkt6MWUrKJLA2875Z6hXA1UF5MpY78P/fLjEtvMGCF0RRO7mv/I1EwGLao+AG/SkuVONFI8xE//aOKV4y/jm3xnimYSWawBXfnqeQNE+ZRs/T1NNP97jHmUWanqLDpbR6eDWXj2GY5heYL8w/r9ozypkns/G8yKpcy7fcGXwm0PI1scEikU+tw0G5G7nOxwGTqqMEhXgWNyah6q0z2rsaXTMxHyjxmVvxPYMU= ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968 2024-05-04T12:09:07.532 DEBUG:teuthology.orchestra.run.smithi146:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCm7xcLdyPGUWO9LQ/mT1Bj8RF6DufxLVkza6YzpUq4GneHJDREQmAW/JWfepuD22rTR9AYtSiKp/G0PdhjHlVIf7A6NoWr3zuQFlBqe18A8sVglmLdRqex/E7LXYvPES34t8AmyLv+VOhRQ7gami3ArHe5RUV/C8XVjwksjitTyixV9Hl4V36A2vDWpkem49t0gMYY8OIOSRPBkq+P2KROawCZcONyfYApVbDL1548FIzbzfz3Lhlbok4PrOMs5TKkbw+/N+PT8NTB+wxpxN79gBYXoKQkt6MWUrKJLA2875Z6hXA1UF5MpY78P/fLjEtvMGCF0RRO7mv/I1EwGLao+AG/SkuVONFI8xE//aOKV4y/jm3xnimYSWawBXfnqeQNE+ZRs/T1NNP97jHmUWanqLDpbR6eDWXj2GY5heYL8w/r9ozypkns/G8yKpcy7fcGXwm0PI1scEikU+tw0G5G7nOxwGTqqMEhXgWNyah6q0z2rsaXTMxHyjxmVvxPYMU= ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-05-04T12:09:07.571 INFO:teuthology.orchestra.run.smithi146.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCm7xcLdyPGUWO9LQ/mT1Bj8RF6DufxLVkza6YzpUq4GneHJDREQmAW/JWfepuD22rTR9AYtSiKp/G0PdhjHlVIf7A6NoWr3zuQFlBqe18A8sVglmLdRqex/E7LXYvPES34t8AmyLv+VOhRQ7gami3ArHe5RUV/C8XVjwksjitTyixV9Hl4V36A2vDWpkem49t0gMYY8OIOSRPBkq+P2KROawCZcONyfYApVbDL1548FIzbzfz3Lhlbok4PrOMs5TKkbw+/N+PT8NTB+wxpxN79gBYXoKQkt6MWUrKJLA2875Z6hXA1UF5MpY78P/fLjEtvMGCF0RRO7mv/I1EwGLao+AG/SkuVONFI8xE//aOKV4y/jm3xnimYSWawBXfnqeQNE+ZRs/T1NNP97jHmUWanqLDpbR6eDWXj2GY5heYL8w/r9ozypkns/G8yKpcy7fcGXwm0PI1scEikU+tw0G5G7nOxwGTqqMEhXgWNyah6q0z2rsaXTMxHyjxmVvxPYMU= ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968 2024-05-04T12:09:07.583 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d9b985c4-0a0e-11ef-bc97-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-05-04T12:09:07.777 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:09:07 smithi079 ceph-mon[27222]: from='client.? 172.21.15.79:0/3929272773' entity='client.admin' 2024-05-04T12:09:07.864 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/d9b985c4-0a0e-11ef-bc97-c7b262605968/mon.smithi079/config 2024-05-04T12:09:10.303 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:10.302+0000 7f6c3e13c700 1 -- 172.21.15.79:0/2973583440 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6c380ff780 msgr2=0x7f6c380ffba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:09:10.303 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:10.302+0000 7f6c3e13c700 1 --2- 172.21.15.79:0/2973583440 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6c380ff780 0x7f6c380ffba0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f6c28009910 tx=0x7f6c28009c20 comp rx=0 tx=0).stop 2024-05-04T12:09:10.303 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:10.303+0000 7f6c3e13c700 1 -- 172.21.15.79:0/2973583440 shutdown_connections 2024-05-04T12:09:10.303 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:10.303+0000 7f6c3e13c700 1 --2- 172.21.15.79:0/2973583440 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6c380ff780 0x7f6c380ffba0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:09:10.303 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:10.303+0000 7f6c3e13c700 1 -- 172.21.15.79:0/2973583440 >> 172.21.15.79:0/2973583440 conn(0x7f6c380fad20 msgr2=0x7f6c380fd160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:09:10.303 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:10.303+0000 7f6c3e13c700 1 -- 172.21.15.79:0/2973583440 shutdown_connections 2024-05-04T12:09:10.304 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:10.303+0000 7f6c3e13c700 1 -- 172.21.15.79:0/2973583440 wait complete. 2024-05-04T12:09:10.304 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:10.303+0000 7f6c3e13c700 1 Processor -- start 2024-05-04T12:09:10.304 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:10.303+0000 7f6c3e13c700 1 -- start start 2024-05-04T12:09:10.304 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:10.304+0000 7f6c3e13c700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6c380ff780 0x7f6c38068b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:09:10.304 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:10.304+0000 7f6c3e13c700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c381986d0 con 0x7f6c380ff780 2024-05-04T12:09:10.304 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:10.304+0000 7f6c2f7fe700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6c380ff780 0x7f6c38068b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:09:10.304 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:10.304+0000 7f6c2f7fe700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6c380ff780 0x7f6c38068b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:55466/0 (socket says 172.21.15.79:55466) 2024-05-04T12:09:10.304 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:10.304+0000 7f6c2f7fe700 1 -- 172.21.15.79:0/2598826040 learned_addr learned my addr 172.21.15.79:0/2598826040 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:09:10.304 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:10.304+0000 7f6c2f7fe700 1 -- 172.21.15.79:0/2598826040 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c280095f0 con 0x7f6c380ff780 2024-05-04T12:09:10.304 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:10.304+0000 7f6c2f7fe700 1 --2- 172.21.15.79:0/2598826040 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6c380ff780 0x7f6c38068b10 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f6c28005b40 tx=0x7f6c28004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:09:10.305 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:10.304+0000 7f6c357fa700 1 -- 172.21.15.79:0/2598826040 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6c2801d070 con 0x7f6c380ff780 2024-05-04T12:09:10.306 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:10.304+0000 7f6c357fa700 1 -- 172.21.15.79:0/2598826040 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f6c2800be10 con 0x7f6c380ff780 2024-05-04T12:09:10.306 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:10.304+0000 7f6c3e13c700 1 -- 172.21.15.79:0/2598826040 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c38198810 con 0x7f6c380ff780 2024-05-04T12:09:10.306 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:10.305+0000 7f6c357fa700 1 -- 172.21.15.79:0/2598826040 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6c2800f460 con 0x7f6c380ff780 2024-05-04T12:09:10.306 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:10.305+0000 7f6c3e13c700 1 -- 172.21.15.79:0/2598826040 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c38198cb0 con 0x7f6c380ff780 2024-05-04T12:09:10.306 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:10.305+0000 7f6c357fa700 1 -- 172.21.15.79:0/2598826040 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 13) v1 ==== 45339+0+0 (secure 0 0 0) 0x7f6c280227e0 con 0x7f6c380ff780 2024-05-04T12:09:10.306 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:10.306+0000 7f6c3e13c700 1 -- 172.21.15.79:0/2598826040 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c3818e3a0 con 0x7f6c380ff780 2024-05-04T12:09:10.306 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:10.306+0000 7f6c357fa700 1 --2- 172.21.15.79:0/2598826040 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7f6c24038410 0x7f6c2403a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:09:10.306 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:10.306+0000 7f6c357fa700 1 -- 172.21.15.79:0/2598826040 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f6c2804c360 con 0x7f6c380ff780 2024-05-04T12:09:10.306 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:10.306+0000 7f6c377fe700 1 --2- 172.21.15.79:0/2598826040 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7f6c24038410 0x7f6c2403a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:09:10.306 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:10.306+0000 7f6c377fe700 1 --2- 172.21.15.79:0/2598826040 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7f6c24038410 0x7f6c2403a8d0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f6c20006fd0 tx=0x7f6c20006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:09:10.309 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:10.310+0000 7f6c357fa700 1 -- 172.21.15.79:0/2598826040 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f6c2802ba00 con 0x7f6c380ff780 2024-05-04T12:09:10.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:09:10 smithi079 ceph-mon[27222]: from='mgr.14164 172.21.15.79:0/395052435' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:09:10.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:09:10 smithi079 ceph-mon[27222]: from='mgr.14164 172.21.15.79:0/395052435' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:09:10.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:09:10 smithi079 ceph-mon[27222]: from='mgr.14164 172.21.15.79:0/395052435' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config rm", "who": "osd/host:smithi079", "name": "osd_memory_target"}]: dispatch 2024-05-04T12:09:10.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:09:10 smithi079 ceph-mon[27222]: from='mgr.14164 172.21.15.79:0/395052435' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:09:10.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:09:10 smithi079 ceph-mon[27222]: from='mgr.14164 172.21.15.79:0/395052435' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi079", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-05-04T12:09:10.446 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:09:10 smithi079 ceph-mon[27222]: from='mgr.14164 172.21.15.79:0/395052435' entity='mgr.smithi079.dgcpwl' cmd='[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi079", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]': finished 2024-05-04T12:09:10.446 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:09:10 smithi079 ceph-mon[27222]: from='mgr.14164 172.21.15.79:0/395052435' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-04T12:09:10.446 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:09:10 smithi079 ceph-mon[27222]: Deploying daemon ceph-exporter.smithi079 on smithi079 2024-05-04T12:09:10.505 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:10.506+0000 7f6c3e13c700 1 -- 172.21.15.79:0/2598826040 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7f6c3802cc70 con 0x7f6c380ff780 2024-05-04T12:09:10.509 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:10.510+0000 7f6c357fa700 1 -- 172.21.15.79:0/2598826040 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7f6c28027020 con 0x7f6c380ff780 2024-05-04T12:09:10.512 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:10.513+0000 7f6c3e13c700 1 -- 172.21.15.79:0/2598826040 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7f6c24038410 msgr2=0x7f6c2403a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:09:10.513 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:10.513+0000 7f6c3e13c700 1 --2- 172.21.15.79:0/2598826040 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7f6c24038410 0x7f6c2403a8d0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f6c20006fd0 tx=0x7f6c20006e40 comp rx=0 tx=0).stop 2024-05-04T12:09:10.513 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:10.513+0000 7f6c3e13c700 1 -- 172.21.15.79:0/2598826040 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6c380ff780 msgr2=0x7f6c38068b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:09:10.513 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:10.513+0000 7f6c3e13c700 1 --2- 172.21.15.79:0/2598826040 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6c380ff780 0x7f6c38068b10 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f6c28005b40 tx=0x7f6c28004740 comp rx=0 tx=0).stop 2024-05-04T12:09:10.513 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:10.513+0000 7f6c3e13c700 1 -- 172.21.15.79:0/2598826040 shutdown_connections 2024-05-04T12:09:10.513 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:10.513+0000 7f6c3e13c700 1 --2- 172.21.15.79:0/2598826040 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7f6c24038410 0x7f6c2403a8d0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:09:10.513 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:10.513+0000 7f6c3e13c700 1 --2- 172.21.15.79:0/2598826040 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6c380ff780 0x7f6c38068b10 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:09:10.513 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:10.513+0000 7f6c3e13c700 1 -- 172.21.15.79:0/2598826040 >> 172.21.15.79:0/2598826040 conn(0x7f6c380fad20 msgr2=0x7f6c380fba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:09:10.513 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:10.513+0000 7f6c3e13c700 1 -- 172.21.15.79:0/2598826040 shutdown_connections 2024-05-04T12:09:10.513 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:10.513+0000 7f6c3e13c700 1 -- 172.21.15.79:0/2598826040 wait complete. 2024-05-04T12:09:11.236 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-05-04T12:09:11.236 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d9b985c4-0a0e-11ef-bc97-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-05-04T12:09:11.833 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:09:11 smithi079 ceph-mon[27222]: from='client.? 172.21.15.79:0/2598826040' entity='client.admin' 2024-05-04T12:09:12.303 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/d9b985c4-0a0e-11ef-bc97-c7b262605968/mon.smithi079/config 2024-05-04T12:09:14.153 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:14.152+0000 7f196b1fc700 1 -- 172.21.15.79:0/3230630381 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f19640ff780 msgr2=0x7f19640ffba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:09:14.153 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:14.152+0000 7f196b1fc700 1 --2- 172.21.15.79:0/3230630381 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f19640ff780 0x7f19640ffba0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f1958009910 tx=0x7f1958009c20 comp rx=0 tx=0).stop 2024-05-04T12:09:14.153 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:14.152+0000 7f196b1fc700 1 -- 172.21.15.79:0/3230630381 shutdown_connections 2024-05-04T12:09:14.153 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:14.152+0000 7f196b1fc700 1 --2- 172.21.15.79:0/3230630381 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f19640ff780 0x7f19640ffba0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:09:14.153 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:14.152+0000 7f196b1fc700 1 -- 172.21.15.79:0/3230630381 >> 172.21.15.79:0/3230630381 conn(0x7f19640fad00 msgr2=0x7f19640fd160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:09:14.154 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:14.152+0000 7f196b1fc700 1 -- 172.21.15.79:0/3230630381 shutdown_connections 2024-05-04T12:09:14.154 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:14.152+0000 7f196b1fc700 1 -- 172.21.15.79:0/3230630381 wait complete. 2024-05-04T12:09:14.154 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:14.153+0000 7f196b1fc700 1 Processor -- start 2024-05-04T12:09:14.154 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:14.153+0000 7f196b1fc700 1 -- start start 2024-05-04T12:09:14.154 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:14.153+0000 7f196b1fc700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f19640ff780 0x7f1964069380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:09:14.154 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:14.153+0000 7f196b1fc700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1964196510 con 0x7f19640ff780 2024-05-04T12:09:14.154 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:14.153+0000 7f1968f98700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f19640ff780 0x7f1964069380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:09:14.154 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:14.153+0000 7f1968f98700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f19640ff780 0x7f1964069380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:36580/0 (socket says 172.21.15.79:36580) 2024-05-04T12:09:14.154 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:14.153+0000 7f1968f98700 1 -- 172.21.15.79:0/3520435867 learned_addr learned my addr 172.21.15.79:0/3520435867 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:09:14.154 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:14.153+0000 7f1968f98700 1 -- 172.21.15.79:0/3520435867 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19580095f0 con 0x7f19640ff780 2024-05-04T12:09:14.154 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:14.153+0000 7f1968f98700 1 --2- 172.21.15.79:0/3520435867 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f19640ff780 0x7f1964069380 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f1958005b40 tx=0x7f1958004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:09:14.154 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:14.154+0000 7f1961ffb700 1 -- 172.21.15.79:0/3520435867 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f195801d070 con 0x7f19640ff780 2024-05-04T12:09:14.154 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:14.154+0000 7f1961ffb700 1 -- 172.21.15.79:0/3520435867 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f195800be10 con 0x7f19640ff780 2024-05-04T12:09:14.154 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:14.154+0000 7f196b1fc700 1 -- 172.21.15.79:0/3520435867 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f196419ab20 con 0x7f19640ff780 2024-05-04T12:09:14.155 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:14.154+0000 7f1961ffb700 1 -- 172.21.15.79:0/3520435867 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f195800f460 con 0x7f19640ff780 2024-05-04T12:09:14.155 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:14.154+0000 7f196b1fc700 1 -- 172.21.15.79:0/3520435867 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f196419ae80 con 0x7f19640ff780 2024-05-04T12:09:14.155 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:14.155+0000 7f1961ffb700 1 -- 172.21.15.79:0/3520435867 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 13) v1 ==== 45339+0+0 (secure 0 0 0) 0x7f195800f6d0 con 0x7f19640ff780 2024-05-04T12:09:14.156 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:14.155+0000 7f196b1fc700 1 -- 172.21.15.79:0/3520435867 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f196418e3a0 con 0x7f19640ff780 2024-05-04T12:09:14.156 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:14.155+0000 7f1961ffb700 1 --2- 172.21.15.79:0/3520435867 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7f19540383c0 0x7f195403a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:09:14.156 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:14.155+0000 7f1961ffb700 1 -- 172.21.15.79:0/3520435867 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f195804c490 con 0x7f19640ff780 2024-05-04T12:09:14.156 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:14.155+0000 7f1963fff700 1 --2- 172.21.15.79:0/3520435867 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7f19540383c0 0x7f195403a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:09:14.156 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:14.156+0000 7f1963fff700 1 --2- 172.21.15.79:0/3520435867 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7f19540383c0 0x7f195403a880 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f1950006fd0 tx=0x7f1950006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:09:14.159 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:14.160+0000 7f1961ffb700 1 -- 172.21.15.79:0/3520435867 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f195802a360 con 0x7f19640ff780 2024-05-04T12:09:14.377 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:14.378+0000 7f196b1fc700 1 -- 172.21.15.79:0/3520435867 --> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7f19640611d0 con 0x7f19540383c0 2024-05-04T12:09:14.380 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:14.382+0000 7f1961ffb700 1 -- 172.21.15.79:0/3520435867 <== mgr.14164 v2:172.21.15.79:6800/4263681324 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f19640611d0 con 0x7f19540383c0 2024-05-04T12:09:14.383 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:14.384+0000 7f196b1fc700 1 -- 172.21.15.79:0/3520435867 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7f19540383c0 msgr2=0x7f195403a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:09:14.383 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:14.384+0000 7f196b1fc700 1 --2- 172.21.15.79:0/3520435867 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7f19540383c0 0x7f195403a880 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f1950006fd0 tx=0x7f1950006e40 comp rx=0 tx=0).stop 2024-05-04T12:09:14.383 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:14.384+0000 7f196b1fc700 1 -- 172.21.15.79:0/3520435867 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f19640ff780 msgr2=0x7f1964069380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:09:14.383 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:14.384+0000 7f196b1fc700 1 --2- 172.21.15.79:0/3520435867 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f19640ff780 0x7f1964069380 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f1958005b40 tx=0x7f1958004740 comp rx=0 tx=0).stop 2024-05-04T12:09:14.383 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:14.384+0000 7f196b1fc700 1 -- 172.21.15.79:0/3520435867 shutdown_connections 2024-05-04T12:09:14.383 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:14.384+0000 7f196b1fc700 1 --2- 172.21.15.79:0/3520435867 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7f19540383c0 0x7f195403a880 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:09:14.383 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:14.384+0000 7f196b1fc700 1 --2- 172.21.15.79:0/3520435867 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f19640ff780 0x7f1964069380 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:09:14.383 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:14.384+0000 7f196b1fc700 1 -- 172.21.15.79:0/3520435867 >> 172.21.15.79:0/3520435867 conn(0x7f19640fad00 msgr2=0x7f19640fb9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:09:14.383 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:14.384+0000 7f196b1fc700 1 -- 172.21.15.79:0/3520435867 shutdown_connections 2024-05-04T12:09:14.383 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:14.384+0000 7f196b1fc700 1 -- 172.21.15.79:0/3520435867 wait complete. 2024-05-04T12:09:15.047 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi146 2024-05-04T12:09:15.047 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-05-04T12:09:15.047 DEBUG:teuthology.orchestra.run.smithi146:> dd of=/etc/ceph/ceph.conf 2024-05-04T12:09:15.065 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-05-04T12:09:15.066 DEBUG:teuthology.orchestra.run.smithi146:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-05-04T12:09:15.124 INFO:tasks.cephadm:Adding host smithi146 to orchestrator... 2024-05-04T12:09:15.124 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d9b985c4-0a0e-11ef-bc97-c7b262605968 -- ceph orch host add smithi146 2024-05-04T12:09:15.334 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/d9b985c4-0a0e-11ef-bc97-c7b262605968/mon.smithi079/config 2024-05-04T12:09:15.695 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:09:15 smithi079 ceph-mon[27222]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-05-04T12:09:15.695 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:09:15 smithi079 ceph-mon[27222]: from='mgr.14164 172.21.15.79:0/395052435' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:09:15.695 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:09:15 smithi079 ceph-mon[27222]: from='mgr.14164 172.21.15.79:0/395052435' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:09:15.695 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:09:15 smithi079 ceph-mon[27222]: from='mgr.14164 172.21.15.79:0/395052435' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:09:15.695 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:09:15 smithi079 ceph-mon[27222]: from='mgr.14164 172.21.15.79:0/395052435' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:09:15.695 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:09:15 smithi079 ceph-mon[27222]: from='mgr.14164 172.21.15.79:0/395052435' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:09:15.695 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:09:15 smithi079 ceph-mon[27222]: from='mgr.14164 172.21.15.79:0/395052435' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi079", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-05-04T12:09:15.695 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:09:15 smithi079 ceph-mon[27222]: from='mgr.14164 172.21.15.79:0/395052435' entity='mgr.smithi079.dgcpwl' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi079", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-05-04T12:09:15.695 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:09:15 smithi079 ceph-mon[27222]: from='mgr.14164 172.21.15.79:0/395052435' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-04T12:09:16.592 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:09:16 smithi079 ceph-mon[27222]: Deploying daemon crash.smithi079 on smithi079 2024-05-04T12:09:16.789 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:16.788+0000 7fc91e13b700 1 -- 172.21.15.79:0/2514587739 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc9180ff780 msgr2=0x7fc9180ffba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:09:16.789 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:16.788+0000 7fc91e13b700 1 --2- 172.21.15.79:0/2514587739 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc9180ff780 0x7fc9180ffba0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fc908009910 tx=0x7fc908009c20 comp rx=0 tx=0).stop 2024-05-04T12:09:16.789 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:16.788+0000 7fc91e13b700 1 -- 172.21.15.79:0/2514587739 shutdown_connections 2024-05-04T12:09:16.789 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:16.788+0000 7fc91e13b700 1 --2- 172.21.15.79:0/2514587739 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc9180ff780 0x7fc9180ffba0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:09:16.789 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:16.788+0000 7fc91e13b700 1 -- 172.21.15.79:0/2514587739 >> 172.21.15.79:0/2514587739 conn(0x7fc9180fad20 msgr2=0x7fc9180fd160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:09:16.789 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:16.788+0000 7fc91e13b700 1 -- 172.21.15.79:0/2514587739 shutdown_connections 2024-05-04T12:09:16.789 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:16.788+0000 7fc91e13b700 1 -- 172.21.15.79:0/2514587739 wait complete. 2024-05-04T12:09:16.789 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:16.789+0000 7fc91e13b700 1 Processor -- start 2024-05-04T12:09:16.789 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:16.789+0000 7fc91e13b700 1 -- start start 2024-05-04T12:09:16.789 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:16.789+0000 7fc91e13b700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc9180ff780 0x7fc9181948a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:09:16.789 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:16.789+0000 7fc91e13b700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc918194de0 con 0x7fc9180ff780 2024-05-04T12:09:16.789 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:16.789+0000 7fc9177fe700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc9180ff780 0x7fc9181948a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:09:16.790 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:16.789+0000 7fc9177fe700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc9180ff780 0x7fc9181948a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:36600/0 (socket says 172.21.15.79:36600) 2024-05-04T12:09:16.790 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:16.789+0000 7fc9177fe700 1 -- 172.21.15.79:0/913835614 learned_addr learned my addr 172.21.15.79:0/913835614 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:09:16.790 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:16.790+0000 7fc9177fe700 1 -- 172.21.15.79:0/913835614 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9080095f0 con 0x7fc9180ff780 2024-05-04T12:09:16.790 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:16.790+0000 7fc9177fe700 1 --2- 172.21.15.79:0/913835614 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc9180ff780 0x7fc9181948a0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fc908005950 tx=0x7fc908009e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:09:16.790 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:16.790+0000 7fc914ff9700 1 -- 172.21.15.79:0/913835614 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc90801d070 con 0x7fc9180ff780 2024-05-04T12:09:16.790 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:16.790+0000 7fc91e13b700 1 -- 172.21.15.79:0/913835614 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc918194fe0 con 0x7fc9180ff780 2024-05-04T12:09:16.790 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:16.790+0000 7fc914ff9700 1 -- 172.21.15.79:0/913835614 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fc90800be10 con 0x7fc9180ff780 2024-05-04T12:09:16.790 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:16.790+0000 7fc914ff9700 1 -- 172.21.15.79:0/913835614 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc90800f460 con 0x7fc9180ff780 2024-05-04T12:09:16.790 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:16.790+0000 7fc91e13b700 1 -- 172.21.15.79:0/913835614 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc918195480 con 0x7fc9180ff780 2024-05-04T12:09:16.791 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:16.791+0000 7fc914ff9700 1 -- 172.21.15.79:0/913835614 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 13) v1 ==== 45339+0+0 (secure 0 0 0) 0x7fc90800f690 con 0x7fc9180ff780 2024-05-04T12:09:16.791 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:16.791+0000 7fc91e13b700 1 -- 172.21.15.79:0/913835614 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc91818e3a0 con 0x7fc9180ff780 2024-05-04T12:09:16.791 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:16.791+0000 7fc914ff9700 1 --2- 172.21.15.79:0/913835614 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7fc9000383c0 0x7fc90003a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:09:16.791 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:16.791+0000 7fc914ff9700 1 -- 172.21.15.79:0/913835614 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fc90804c500 con 0x7fc9180ff780 2024-05-04T12:09:16.791 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:16.791+0000 7fc916ffd700 1 --2- 172.21.15.79:0/913835614 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7fc9000383c0 0x7fc90003a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:09:16.791 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:16.791+0000 7fc916ffd700 1 --2- 172.21.15.79:0/913835614 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7fc9000383c0 0x7fc90003a880 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fc90c006fd0 tx=0x7fc90c006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:09:16.794 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:16.796+0000 7fc914ff9700 1 -- 172.21.15.79:0/913835614 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fc908020550 con 0x7fc9180ff780 2024-05-04T12:09:16.989 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:16.990+0000 7fc91e13b700 1 -- 172.21.15.79:0/913835614 --> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi146", "target": ["mon-mgr", ""]}) v1 -- 0x7fc9180611d0 con 0x7fc9000383c0 2024-05-04T12:09:17.917 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:09:17 smithi079 ceph-mon[27222]: from='client.14191 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi146", "target": ["mon-mgr", ""]}]: dispatch 2024-05-04T12:09:19.457 INFO:teuthology.orchestra.run.smithi079.stdout:Added host 'smithi146' with addr '172.21.15.146' 2024-05-04T12:09:19.457 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:19.457+0000 7fc914ff9700 1 -- 172.21.15.79:0/913835614 <== mgr.14164 v2:172.21.15.79:6800/4263681324 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7fc9180611d0 con 0x7fc9000383c0 2024-05-04T12:09:19.458 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:19.459+0000 7fc91e13b700 1 -- 172.21.15.79:0/913835614 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7fc9000383c0 msgr2=0x7fc90003a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:09:19.458 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:19.459+0000 7fc91e13b700 1 --2- 172.21.15.79:0/913835614 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7fc9000383c0 0x7fc90003a880 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fc90c006fd0 tx=0x7fc90c006e40 comp rx=0 tx=0).stop 2024-05-04T12:09:19.459 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:19.459+0000 7fc91e13b700 1 -- 172.21.15.79:0/913835614 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc9180ff780 msgr2=0x7fc9181948a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:09:19.459 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:19.459+0000 7fc91e13b700 1 --2- 172.21.15.79:0/913835614 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc9180ff780 0x7fc9181948a0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fc908005950 tx=0x7fc908009e60 comp rx=0 tx=0).stop 2024-05-04T12:09:19.459 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:19.459+0000 7fc91e13b700 1 -- 172.21.15.79:0/913835614 shutdown_connections 2024-05-04T12:09:19.459 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:19.459+0000 7fc91e13b700 1 --2- 172.21.15.79:0/913835614 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7fc9000383c0 0x7fc90003a880 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:09:19.459 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:19.459+0000 7fc91e13b700 1 --2- 172.21.15.79:0/913835614 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc9180ff780 0x7fc9181948a0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:09:19.459 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:19.459+0000 7fc91e13b700 1 -- 172.21.15.79:0/913835614 >> 172.21.15.79:0/913835614 conn(0x7fc9180fad20 msgr2=0x7fc9180fba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:09:19.459 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:19.460+0000 7fc91e13b700 1 -- 172.21.15.79:0/913835614 shutdown_connections 2024-05-04T12:09:19.459 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:19.460+0000 7fc91e13b700 1 -- 172.21.15.79:0/913835614 wait complete. 2024-05-04T12:09:19.662 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:09:19 smithi079 ceph-mon[27222]: Deploying cephadm binary to smithi146 2024-05-04T12:09:19.662 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:09:19 smithi079 ceph-mon[27222]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-04T12:09:19.663 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:09:19 smithi079 ceph-mon[27222]: from='mgr.14164 172.21.15.79:0/395052435' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:09:19.663 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:09:19 smithi079 ceph-mon[27222]: from='mgr.14164 172.21.15.79:0/395052435' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:09:19.663 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:09:19 smithi079 ceph-mon[27222]: from='mgr.14164 172.21.15.79:0/395052435' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:09:19.663 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:09:19 smithi079 ceph-mon[27222]: from='mgr.14164 172.21.15.79:0/395052435' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:09:19.663 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:09:19 smithi079 ceph-mon[27222]: from='mgr.14164 172.21.15.79:0/395052435' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:09:20.015 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d9b985c4-0a0e-11ef-bc97-c7b262605968 -- ceph orch host ls --format=json 2024-05-04T12:09:20.227 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/d9b985c4-0a0e-11ef-bc97-c7b262605968/mon.smithi079/config 2024-05-04T12:09:20.647 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:09:20 smithi079 ceph-mon[27222]: Deploying daemon node-exporter.smithi079 on smithi079 2024-05-04T12:09:20.647 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:09:20 smithi079 ceph-mon[27222]: from='mgr.14164 172.21.15.79:0/395052435' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:09:20.647 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:09:20 smithi079 ceph-mon[27222]: Added host smithi146 2024-05-04T12:09:21.695 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:09:21 smithi079 ceph-mon[27222]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-04T12:09:22.334 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:22.328+0000 7f97c78b6700 1 -- 172.21.15.79:0/3177423541 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f97c00ff780 msgr2=0x7f97c00ffba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:09:22.335 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:22.328+0000 7f97c78b6700 1 --2- 172.21.15.79:0/3177423541 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f97c00ff780 0x7f97c00ffba0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f97b8009910 tx=0x7f97b8009c20 comp rx=0 tx=0).stop 2024-05-04T12:09:22.335 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:22.328+0000 7f97c78b6700 1 -- 172.21.15.79:0/3177423541 shutdown_connections 2024-05-04T12:09:22.335 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:22.328+0000 7f97c78b6700 1 --2- 172.21.15.79:0/3177423541 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f97c00ff780 0x7f97c00ffba0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:09:22.335 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:22.328+0000 7f97c78b6700 1 -- 172.21.15.79:0/3177423541 >> 172.21.15.79:0/3177423541 conn(0x7f97c00fad00 msgr2=0x7f97c00fd160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:09:22.335 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:22.329+0000 7f97c78b6700 1 -- 172.21.15.79:0/3177423541 shutdown_connections 2024-05-04T12:09:22.335 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:22.329+0000 7f97c78b6700 1 -- 172.21.15.79:0/3177423541 wait complete. 2024-05-04T12:09:22.335 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:22.329+0000 7f97c78b6700 1 Processor -- start 2024-05-04T12:09:22.335 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:22.329+0000 7f97c78b6700 1 -- start start 2024-05-04T12:09:22.335 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:22.329+0000 7f97c78b6700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f97c00ff780 0x7f97c0197bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:09:22.335 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:22.330+0000 7f97c78b6700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97c0198100 con 0x7f97c00ff780 2024-05-04T12:09:22.335 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:22.330+0000 7f97c5652700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f97c00ff780 0x7f97c0197bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:09:22.335 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:22.330+0000 7f97c5652700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f97c00ff780 0x7f97c0197bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:36614/0 (socket says 172.21.15.79:36614) 2024-05-04T12:09:22.336 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:22.330+0000 7f97c5652700 1 -- 172.21.15.79:0/4109760655 learned_addr learned my addr 172.21.15.79:0/4109760655 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:09:22.336 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:22.330+0000 7f97c5652700 1 -- 172.21.15.79:0/4109760655 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97b80095f0 con 0x7f97c00ff780 2024-05-04T12:09:22.336 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:22.330+0000 7f97c5652700 1 --2- 172.21.15.79:0/4109760655 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f97c00ff780 0x7f97c0197bc0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f97b8005b40 tx=0x7f97b8004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:09:22.336 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:22.330+0000 7f97b67fc700 1 -- 172.21.15.79:0/4109760655 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f97b801d070 con 0x7f97c00ff780 2024-05-04T12:09:22.336 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:22.330+0000 7f97c78b6700 1 -- 172.21.15.79:0/4109760655 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97c0198300 con 0x7f97c00ff780 2024-05-04T12:09:22.336 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:22.330+0000 7f97c78b6700 1 -- 172.21.15.79:0/4109760655 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97c01987a0 con 0x7f97c00ff780 2024-05-04T12:09:22.336 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:22.330+0000 7f97b67fc700 1 -- 172.21.15.79:0/4109760655 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f97b8005c00 con 0x7f97c00ff780 2024-05-04T12:09:22.336 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:22.330+0000 7f97b67fc700 1 -- 172.21.15.79:0/4109760655 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f97b8022c10 con 0x7f97c00ff780 2024-05-04T12:09:22.336 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:22.331+0000 7f97b67fc700 1 -- 172.21.15.79:0/4109760655 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 13) v1 ==== 45339+0+0 (secure 0 0 0) 0x7f97b800f460 con 0x7f97c00ff780 2024-05-04T12:09:22.336 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:22.331+0000 7f97c78b6700 1 -- 172.21.15.79:0/4109760655 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97a4005320 con 0x7f97c00ff780 2024-05-04T12:09:22.336 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:22.332+0000 7f97b67fc700 1 --2- 172.21.15.79:0/4109760655 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7f97ac0383c0 0x7f97ac03a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:09:22.336 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:22.332+0000 7f97b67fc700 1 -- 172.21.15.79:0/4109760655 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f97b804c0d0 con 0x7f97c00ff780 2024-05-04T12:09:22.336 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:22.332+0000 7f97c4e51700 1 --2- 172.21.15.79:0/4109760655 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7f97ac0383c0 0x7f97ac03a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:09:22.336 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:22.332+0000 7f97c4e51700 1 --2- 172.21.15.79:0/4109760655 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7f97ac0383c0 0x7f97ac03a880 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f97b0006fd0 tx=0x7f97b0006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:09:22.339 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:22.341+0000 7f97b67fc700 1 -- 172.21.15.79:0/4109760655 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f97b800f8f0 con 0x7f97c00ff780 2024-05-04T12:09:22.540 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:22.541+0000 7f97c78b6700 1 -- 172.21.15.79:0/4109760655 --> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f97a4000bf0 con 0x7f97ac0383c0 2024-05-04T12:09:22.541 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:22.543+0000 7f97b67fc700 1 -- 172.21.15.79:0/4109760655 <== mgr.14164 v2:172.21.15.79:6800/4263681324 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+160 (secure 0 0 0) 0x7f97a4000bf0 con 0x7f97ac0383c0 2024-05-04T12:09:22.542 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-05-04T12:09:22.542 INFO:teuthology.orchestra.run.smithi079.stdout:[{"addr": "172.21.15.79", "hostname": "smithi079", "labels": [], "status": ""}, {"addr": "172.21.15.146", "hostname": "smithi146", "labels": [], "status": ""}] 2024-05-04T12:09:22.544 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:22.545+0000 7f97c78b6700 1 -- 172.21.15.79:0/4109760655 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7f97ac0383c0 msgr2=0x7f97ac03a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:09:22.544 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:22.545+0000 7f97c78b6700 1 --2- 172.21.15.79:0/4109760655 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7f97ac0383c0 0x7f97ac03a880 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f97b0006fd0 tx=0x7f97b0006e40 comp rx=0 tx=0).stop 2024-05-04T12:09:22.544 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:22.545+0000 7f97c78b6700 1 -- 172.21.15.79:0/4109760655 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f97c00ff780 msgr2=0x7f97c0197bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:09:22.544 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:22.545+0000 7f97c78b6700 1 --2- 172.21.15.79:0/4109760655 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f97c00ff780 0x7f97c0197bc0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f97b8005b40 tx=0x7f97b8004740 comp rx=0 tx=0).stop 2024-05-04T12:09:22.545 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:22.545+0000 7f97c78b6700 1 -- 172.21.15.79:0/4109760655 shutdown_connections 2024-05-04T12:09:22.545 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:22.545+0000 7f97c78b6700 1 --2- 172.21.15.79:0/4109760655 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7f97ac0383c0 0x7f97ac03a880 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:09:22.545 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:22.545+0000 7f97c78b6700 1 --2- 172.21.15.79:0/4109760655 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f97c00ff780 0x7f97c0197bc0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:09:22.545 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:22.545+0000 7f97c78b6700 1 -- 172.21.15.79:0/4109760655 >> 172.21.15.79:0/4109760655 conn(0x7f97c00fad00 msgr2=0x7f97c0069090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:09:22.545 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:22.545+0000 7f97c78b6700 1 -- 172.21.15.79:0/4109760655 shutdown_connections 2024-05-04T12:09:22.545 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:22.545+0000 7f97c78b6700 1 -- 172.21.15.79:0/4109760655 wait complete. 2024-05-04T12:09:23.628 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:09:23 smithi079 ceph-mon[27222]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-04T12:09:23.628 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:09:23 smithi079 ceph-mon[27222]: from='client.14193 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-04T12:09:23.628 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:09:23 smithi079 ceph-mon[27222]: from='mgr.14164 172.21.15.79:0/395052435' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:09:23.649 INFO:tasks.cephadm:Setting crush tunables to default 2024-05-04T12:09:23.649 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d9b985c4-0a0e-11ef-bc97-c7b262605968 -- ceph osd crush tunables default 2024-05-04T12:09:23.932 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/d9b985c4-0a0e-11ef-bc97-c7b262605968/mon.smithi079/config 2024-05-04T12:09:25.086 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:09:24 smithi079 ceph-mon[27222]: from='mgr.14164 172.21.15.79:0/395052435' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:09:25.086 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:09:24 smithi079 ceph-mon[27222]: from='mgr.14164 172.21.15.79:0/395052435' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:09:25.086 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:09:24 smithi079 ceph-mon[27222]: from='mgr.14164 172.21.15.79:0/395052435' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:09:25.086 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:09:24 smithi079 ceph-mon[27222]: from='mgr.14164 172.21.15.79:0/395052435' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:09:25.086 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:09:24 smithi079 ceph-mon[27222]: Deploying daemon alertmanager.smithi079 on smithi079 2024-05-04T12:09:25.087 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:09:24 smithi079 ceph-mon[27222]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-04T12:09:25.734 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:25.475+0000 7f714ebef700 1 -- 172.21.15.79:0/1561764084 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7148100880 msgr2=0x7f7148100ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:09:25.734 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:25.475+0000 7f714ebef700 1 --2- 172.21.15.79:0/1561764084 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7148100880 0x7f7148100ca0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f713c009910 tx=0x7f713c009c20 comp rx=0 tx=0).stop 2024-05-04T12:09:25.735 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:25.475+0000 7f714ebef700 1 -- 172.21.15.79:0/1561764084 shutdown_connections 2024-05-04T12:09:25.735 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:25.475+0000 7f714ebef700 1 --2- 172.21.15.79:0/1561764084 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7148100880 0x7f7148100ca0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:09:25.735 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:25.475+0000 7f714ebef700 1 -- 172.21.15.79:0/1561764084 >> 172.21.15.79:0/1561764084 conn(0x7f71480fbe00 msgr2=0x7f71480fe260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:09:25.735 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:25.475+0000 7f714ebef700 1 -- 172.21.15.79:0/1561764084 shutdown_connections 2024-05-04T12:09:25.735 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:25.475+0000 7f714ebef700 1 -- 172.21.15.79:0/1561764084 wait complete. 2024-05-04T12:09:25.735 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:25.475+0000 7f714ebef700 1 Processor -- start 2024-05-04T12:09:25.735 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:25.475+0000 7f714ebef700 1 -- start start 2024-05-04T12:09:25.735 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:25.476+0000 7f714ebef700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7148100880 0x7f71481959d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:09:25.735 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:25.476+0000 7f714ebef700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7148195f10 con 0x7f7148100880 2024-05-04T12:09:25.735 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:25.476+0000 7f714c98b700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7148100880 0x7f71481959d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:09:25.735 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:25.476+0000 7f714c98b700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7148100880 0x7f71481959d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:48340/0 (socket says 172.21.15.79:48340) 2024-05-04T12:09:25.735 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:25.476+0000 7f714c98b700 1 -- 172.21.15.79:0/699271291 learned_addr learned my addr 172.21.15.79:0/699271291 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:09:25.735 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:25.476+0000 7f714c98b700 1 -- 172.21.15.79:0/699271291 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f713c0095f0 con 0x7f7148100880 2024-05-04T12:09:25.735 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:25.476+0000 7f714c98b700 1 --2- 172.21.15.79:0/699271291 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7148100880 0x7f71481959d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f713c005b40 tx=0x7f713c004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:09:25.736 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:25.476+0000 7f7145ffb700 1 -- 172.21.15.79:0/699271291 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f713c01d070 con 0x7f7148100880 2024-05-04T12:09:25.736 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:25.476+0000 7f7145ffb700 1 -- 172.21.15.79:0/699271291 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f713c00be10 con 0x7f7148100880 2024-05-04T12:09:25.736 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:25.476+0000 7f714ebef700 1 -- 172.21.15.79:0/699271291 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7148196110 con 0x7f7148100880 2024-05-04T12:09:25.736 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:25.476+0000 7f7145ffb700 1 -- 172.21.15.79:0/699271291 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f713c00f460 con 0x7f7148100880 2024-05-04T12:09:25.736 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:25.476+0000 7f714ebef700 1 -- 172.21.15.79:0/699271291 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71481965b0 con 0x7f7148100880 2024-05-04T12:09:25.736 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:25.477+0000 7f7145ffb700 1 -- 172.21.15.79:0/699271291 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 13) v1 ==== 45339+0+0 (secure 0 0 0) 0x7f713c00f5c0 con 0x7f7148100880 2024-05-04T12:09:25.736 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:25.477+0000 7f714ebef700 1 -- 172.21.15.79:0/699271291 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f714818f4a0 con 0x7f7148100880 2024-05-04T12:09:25.736 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:25.477+0000 7f7145ffb700 1 --2- 172.21.15.79:0/699271291 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7f7138038410 0x7f713803a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:09:25.736 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:25.477+0000 7f7145ffb700 1 -- 172.21.15.79:0/699271291 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f713c04c4e0 con 0x7f7148100880 2024-05-04T12:09:25.736 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:25.478+0000 7f7147fff700 1 --2- 172.21.15.79:0/699271291 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7f7138038410 0x7f713803a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:09:25.736 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:25.478+0000 7f7147fff700 1 --2- 172.21.15.79:0/699271291 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7f7138038410 0x7f713803a8d0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f7134006fd0 tx=0x7f7134006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:09:25.736 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:25.482+0000 7f7145ffb700 1 -- 172.21.15.79:0/699271291 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f713c02a360 con 0x7f7148100880 2024-05-04T12:09:25.737 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:25.676+0000 7f714ebef700 1 -- 172.21.15.79:0/699271291 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f714802cc70 con 0x7f7148100880 2024-05-04T12:09:25.977 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:25.978+0000 7f7145ffb700 1 -- 172.21.15.79:0/699271291 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7f713c027070 con 0x7f7148100880 2024-05-04T12:09:25.979 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:25.980+0000 7f714ebef700 1 -- 172.21.15.79:0/699271291 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7f7138038410 msgr2=0x7f713803a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:09:25.979 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:25.980+0000 7f714ebef700 1 --2- 172.21.15.79:0/699271291 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7f7138038410 0x7f713803a8d0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f7134006fd0 tx=0x7f7134006e40 comp rx=0 tx=0).stop 2024-05-04T12:09:25.979 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:25.980+0000 7f714ebef700 1 -- 172.21.15.79:0/699271291 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7148100880 msgr2=0x7f71481959d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:09:25.979 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:25.980+0000 7f714ebef700 1 --2- 172.21.15.79:0/699271291 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7148100880 0x7f71481959d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f713c005b40 tx=0x7f713c004740 comp rx=0 tx=0).stop 2024-05-04T12:09:25.979 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:25.980+0000 7f714ebef700 1 -- 172.21.15.79:0/699271291 shutdown_connections 2024-05-04T12:09:25.979 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:25.980+0000 7f714ebef700 1 --2- 172.21.15.79:0/699271291 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7f7138038410 0x7f713803a8d0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:09:25.979 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:25.980+0000 7f714ebef700 1 --2- 172.21.15.79:0/699271291 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7148100880 0x7f71481959d0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:09:25.979 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:25.980+0000 7f714ebef700 1 -- 172.21.15.79:0/699271291 >> 172.21.15.79:0/699271291 conn(0x7f71480fbe00 msgr2=0x7f71480fcae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:09:25.979 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:25.980+0000 7f714ebef700 1 -- 172.21.15.79:0/699271291 shutdown_connections 2024-05-04T12:09:25.980 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:09:25.980+0000 7f714ebef700 1 -- 172.21.15.79:0/699271291 wait complete. 2024-05-04T12:09:25.980 INFO:teuthology.orchestra.run.smithi079.stderr:adjusted tunables profile to default 2024-05-04T12:09:26.444 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:09:25 smithi079 ceph-mon[27222]: from='client.? 172.21.15.79:0/699271291' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-05-04T12:09:26.913 INFO:tasks.cephadm:Adding mon.smithi079 on smithi079 2024-05-04T12:09:26.913 INFO:tasks.cephadm:Adding mon.smithi146 on smithi146 2024-05-04T12:09:26.913 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d9b985c4-0a0e-11ef-bc97-c7b262605968 -- ceph orch apply mon '2;smithi079:172.21.15.79=smithi079;smithi146:172.21.15.146=smithi146' 2024-05-04T12:09:27.125 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /etc/ceph/ceph.conf 2024-05-04T12:09:27.180 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /etc/ceph/ceph.conf 2024-05-04T12:09:27.194 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:09:26 smithi079 ceph-mon[27222]: from='client.? 172.21.15.79:0/699271291' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-05-04T12:09:27.195 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:09:26 smithi079 ceph-mon[27222]: osdmap e4: 0 total, 0 up, 0 in 2024-05-04T12:09:27.195 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:09:26 smithi079 ceph-mon[27222]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-04T12:09:28.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:28.689+0000 7fb1d61b6700 1 -- 172.21.15.146:0/299652459 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb1d00ff780 msgr2=0x7fb1d00ffba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:09:28.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:28.689+0000 7fb1d61b6700 1 --2- 172.21.15.146:0/299652459 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb1d00ff780 0x7fb1d00ffba0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fb1b8009910 tx=0x7fb1b8009c20 comp rx=0 tx=0).stop 2024-05-04T12:09:28.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:28.689+0000 7fb1d61b6700 1 -- 172.21.15.146:0/299652459 shutdown_connections 2024-05-04T12:09:28.691 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:28.689+0000 7fb1d61b6700 1 --2- 172.21.15.146:0/299652459 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb1d00ff780 0x7fb1d00ffba0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:09:28.691 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:28.689+0000 7fb1d61b6700 1 -- 172.21.15.146:0/299652459 >> 172.21.15.146:0/299652459 conn(0x7fb1d00fad20 msgr2=0x7fb1d00fd160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:09:28.691 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:28.689+0000 7fb1d61b6700 1 -- 172.21.15.146:0/299652459 shutdown_connections 2024-05-04T12:09:28.691 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:28.689+0000 7fb1d61b6700 1 -- 172.21.15.146:0/299652459 wait complete. 2024-05-04T12:09:28.691 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:28.690+0000 7fb1d61b6700 1 Processor -- start 2024-05-04T12:09:28.691 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:28.690+0000 7fb1d61b6700 1 -- start start 2024-05-04T12:09:28.691 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:28.690+0000 7fb1d61b6700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb1d00ff780 0x7fb1d0195980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:09:28.691 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:28.690+0000 7fb1d61b6700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1d0195ec0 con 0x7fb1d00ff780 2024-05-04T12:09:28.691 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:28.690+0000 7fb1cf7fe700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb1d00ff780 0x7fb1d0195980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:09:28.691 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:28.690+0000 7fb1cf7fe700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb1d00ff780 0x7fb1d0195980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.146:36214/0 (socket says 172.21.15.146:36214) 2024-05-04T12:09:28.691 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:28.690+0000 7fb1cf7fe700 1 -- 172.21.15.146:0/1207772765 learned_addr learned my addr 172.21.15.146:0/1207772765 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-05-04T12:09:28.691 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:28.691+0000 7fb1cf7fe700 1 -- 172.21.15.146:0/1207772765 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1b80095f0 con 0x7fb1d00ff780 2024-05-04T12:09:28.691 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:28.691+0000 7fb1cf7fe700 1 --2- 172.21.15.146:0/1207772765 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb1d00ff780 0x7fb1d0195980 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fb1b8005b40 tx=0x7fb1b8004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:09:28.691 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:28.691+0000 7fb1ccff9700 1 -- 172.21.15.146:0/1207772765 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb1b801d070 con 0x7fb1d00ff780 2024-05-04T12:09:28.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:28.691+0000 7fb1ccff9700 1 -- 172.21.15.146:0/1207772765 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fb1b800be10 con 0x7fb1d00ff780 2024-05-04T12:09:28.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:28.691+0000 7fb1d61b6700 1 -- 172.21.15.146:0/1207772765 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1d01960c0 con 0x7fb1d00ff780 2024-05-04T12:09:28.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:28.691+0000 7fb1ccff9700 1 -- 172.21.15.146:0/1207772765 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb1b800f460 con 0x7fb1d00ff780 2024-05-04T12:09:28.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:28.691+0000 7fb1d61b6700 1 -- 172.21.15.146:0/1207772765 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1d0196560 con 0x7fb1d00ff780 2024-05-04T12:09:28.693 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:28.692+0000 7fb1ccff9700 1 -- 172.21.15.146:0/1207772765 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 13) v1 ==== 45339+0+0 (secure 0 0 0) 0x7fb1b800b960 con 0x7fb1d00ff780 2024-05-04T12:09:28.693 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:28.692+0000 7fb1d61b6700 1 -- 172.21.15.146:0/1207772765 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1d018f3b0 con 0x7fb1d00ff780 2024-05-04T12:09:28.693 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:28.692+0000 7fb1ccff9700 1 --2- 172.21.15.146:0/1207772765 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7fb1bc0383c0 0x7fb1bc03a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:09:28.693 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:28.692+0000 7fb1ccff9700 1 -- 172.21.15.146:0/1207772765 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fb1b804c480 con 0x7fb1d00ff780 2024-05-04T12:09:28.693 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:28.692+0000 7fb1ceffd700 1 --2- 172.21.15.146:0/1207772765 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7fb1bc0383c0 0x7fb1bc03a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:09:28.693 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:28.693+0000 7fb1ceffd700 1 --2- 172.21.15.146:0/1207772765 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7fb1bc0383c0 0x7fb1bc03a880 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fb1c0006fd0 tx=0x7fb1c0006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:09:28.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:28.697+0000 7fb1ccff9700 1 -- 172.21.15.146:0/1207772765 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fb1b802a360 con 0x7fb1d00ff780 2024-05-04T12:09:28.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:28.893+0000 7fb1d61b6700 1 -- 172.21.15.146:0/1207772765 --> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi079:172.21.15.79=smithi079;smithi146:172.21.15.146=smithi146", "target": ["mon-mgr", ""]}) v1 -- 0x7fb1d0061340 con 0x7fb1bc0383c0 2024-05-04T12:09:28.899 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:28.900+0000 7fb1ccff9700 1 -- 172.21.15.146:0/1207772765 <== mgr.14164 v2:172.21.15.79:6800/4263681324 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fb1d0061340 con 0x7fb1bc0383c0 2024-05-04T12:09:28.899 INFO:teuthology.orchestra.run.smithi146.stdout:Scheduled mon update... 2024-05-04T12:09:28.901 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:28.902+0000 7fb1d61b6700 1 -- 172.21.15.146:0/1207772765 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7fb1bc0383c0 msgr2=0x7fb1bc03a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:09:28.901 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:28.902+0000 7fb1d61b6700 1 --2- 172.21.15.146:0/1207772765 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7fb1bc0383c0 0x7fb1bc03a880 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fb1c0006fd0 tx=0x7fb1c0006e40 comp rx=0 tx=0).stop 2024-05-04T12:09:28.901 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:28.902+0000 7fb1d61b6700 1 -- 172.21.15.146:0/1207772765 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb1d00ff780 msgr2=0x7fb1d0195980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:09:28.901 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:28.902+0000 7fb1d61b6700 1 --2- 172.21.15.146:0/1207772765 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb1d00ff780 0x7fb1d0195980 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fb1b8005b40 tx=0x7fb1b8004740 comp rx=0 tx=0).stop 2024-05-04T12:09:28.901 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:28.902+0000 7fb1d61b6700 1 -- 172.21.15.146:0/1207772765 shutdown_connections 2024-05-04T12:09:28.901 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:28.902+0000 7fb1d61b6700 1 --2- 172.21.15.146:0/1207772765 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7fb1bc0383c0 0x7fb1bc03a880 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:09:28.901 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:28.902+0000 7fb1d61b6700 1 --2- 172.21.15.146:0/1207772765 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb1d00ff780 0x7fb1d0195980 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:09:28.901 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:28.902+0000 7fb1d61b6700 1 -- 172.21.15.146:0/1207772765 >> 172.21.15.146:0/1207772765 conn(0x7fb1d00fad20 msgr2=0x7fb1d00690f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:09:28.902 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:28.902+0000 7fb1d61b6700 1 -- 172.21.15.146:0/1207772765 shutdown_connections 2024-05-04T12:09:28.902 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:28.902+0000 7fb1d61b6700 1 -- 172.21.15.146:0/1207772765 wait complete. 2024-05-04T12:09:29.430 DEBUG:teuthology.orchestra.run.smithi146:mon.smithi146> sudo journalctl -f -n 0 -u ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968@mon.smithi146.service 2024-05-04T12:09:29.435 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-05-04T12:09:29.435 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d9b985c4-0a0e-11ef-bc97-c7b262605968 -- ceph mon dump -f json 2024-05-04T12:09:29.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:09:29 smithi079 ceph-mon[27222]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-04T12:09:29.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:09:29 smithi079 ceph-mon[27222]: from='mgr.14164 172.21.15.79:0/395052435' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:09:29.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:09:29 smithi079 ceph-mon[27222]: from='mgr.14164 172.21.15.79:0/395052435' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:09:29.657 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /etc/ceph/ceph.conf 2024-05-04T12:09:29.702 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /etc/ceph/ceph.conf 2024-05-04T12:09:30.408 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:09:30 smithi079 ceph-mon[27222]: from='client.14197 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi079:172.21.15.79=smithi079;smithi146:172.21.15.146=smithi146", "target": ["mon-mgr", ""]}]: dispatch 2024-05-04T12:09:30.408 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:09:30 smithi079 ceph-mon[27222]: Saving service mon spec with placement smithi079:172.21.15.79=smithi079;smithi146:172.21.15.146=smithi146;count:2 2024-05-04T12:09:31.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:31.152+0000 7f6b772bf700 1 -- 172.21.15.146:0/4188288115 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6b701008a0 msgr2=0x7f6b70100cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:09:31.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:31.152+0000 7f6b772bf700 1 --2- 172.21.15.146:0/4188288115 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6b701008a0 0x7f6b70100cc0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f6b60009910 tx=0x7f6b60009c20 comp rx=0 tx=0).stop 2024-05-04T12:09:31.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:31.152+0000 7f6b772bf700 1 -- 172.21.15.146:0/4188288115 shutdown_connections 2024-05-04T12:09:31.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:31.152+0000 7f6b772bf700 1 --2- 172.21.15.146:0/4188288115 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6b701008a0 0x7f6b70100cc0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:09:31.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:31.152+0000 7f6b772bf700 1 -- 172.21.15.146:0/4188288115 >> 172.21.15.146:0/4188288115 conn(0x7f6b700fbe20 msgr2=0x7f6b700fe280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:09:31.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:31.152+0000 7f6b772bf700 1 -- 172.21.15.146:0/4188288115 shutdown_connections 2024-05-04T12:09:31.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:31.152+0000 7f6b772bf700 1 -- 172.21.15.146:0/4188288115 wait complete. 2024-05-04T12:09:31.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:31.153+0000 7f6b772bf700 1 Processor -- start 2024-05-04T12:09:31.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:31.153+0000 7f6b772bf700 1 -- start start 2024-05-04T12:09:31.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:31.153+0000 7f6b772bf700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6b701008a0 0x7f6b701959a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:09:31.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:31.153+0000 7f6b772bf700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b70195ee0 con 0x7f6b701008a0 2024-05-04T12:09:31.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:31.153+0000 7f6b7505b700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6b701008a0 0x7f6b701959a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:09:31.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:31.153+0000 7f6b7505b700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6b701008a0 0x7f6b701959a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.146:36228/0 (socket says 172.21.15.146:36228) 2024-05-04T12:09:31.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:31.153+0000 7f6b7505b700 1 -- 172.21.15.146:0/3297162224 learned_addr learned my addr 172.21.15.146:0/3297162224 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-05-04T12:09:31.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:31.153+0000 7f6b7505b700 1 -- 172.21.15.146:0/3297162224 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b600095f0 con 0x7f6b701008a0 2024-05-04T12:09:31.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:31.154+0000 7f6b7505b700 1 --2- 172.21.15.146:0/3297162224 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6b701008a0 0x7f6b701959a0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f6b60005b40 tx=0x7f6b60004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:09:31.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:31.154+0000 7f6b667fc700 1 -- 172.21.15.146:0/3297162224 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6b6001d070 con 0x7f6b701008a0 2024-05-04T12:09:31.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:31.154+0000 7f6b772bf700 1 -- 172.21.15.146:0/3297162224 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b701960e0 con 0x7f6b701008a0 2024-05-04T12:09:31.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:31.154+0000 7f6b667fc700 1 -- 172.21.15.146:0/3297162224 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f6b6000be10 con 0x7f6b701008a0 2024-05-04T12:09:31.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:31.154+0000 7f6b667fc700 1 -- 172.21.15.146:0/3297162224 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6b6000f460 con 0x7f6b701008a0 2024-05-04T12:09:31.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:31.154+0000 7f6b772bf700 1 -- 172.21.15.146:0/3297162224 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b70196580 con 0x7f6b701008a0 2024-05-04T12:09:31.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:31.155+0000 7f6b772bf700 1 -- 172.21.15.146:0/3297162224 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b7018f4c0 con 0x7f6b701008a0 2024-05-04T12:09:31.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:31.155+0000 7f6b667fc700 1 -- 172.21.15.146:0/3297162224 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 13) v1 ==== 45339+0+0 (secure 0 0 0) 0x7f6b600227e0 con 0x7f6b701008a0 2024-05-04T12:09:31.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:31.155+0000 7f6b667fc700 1 --2- 172.21.15.146:0/3297162224 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7f6b5c0383c0 0x7f6b5c03a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:09:31.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:31.155+0000 7f6b667fc700 1 -- 172.21.15.146:0/3297162224 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f6b6004c560 con 0x7f6b701008a0 2024-05-04T12:09:31.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:31.155+0000 7f6b7485a700 1 --2- 172.21.15.146:0/3297162224 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7f6b5c0383c0 0x7f6b5c03a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:09:31.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:31.156+0000 7f6b7485a700 1 --2- 172.21.15.146:0/3297162224 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7f6b5c0383c0 0x7f6b5c03a880 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f6b6c006fd0 tx=0x7f6b6c006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:09:31.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:31.160+0000 7f6b667fc700 1 -- 172.21.15.146:0/3297162224 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f6b6002bba0 con 0x7f6b701008a0 2024-05-04T12:09:31.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:31.421+0000 7f6b772bf700 1 -- 172.21.15.146:0/3297162224 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6b7002cc70 con 0x7f6b701008a0 2024-05-04T12:09:31.421 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:31.421+0000 7f6b667fc700 1 -- 172.21.15.146:0/3297162224 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f6b60027020 con 0x7f6b701008a0 2024-05-04T12:09:31.421 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-05-04T12:09:31.421 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"d9b985c4-0a0e-11ef-bc97-c7b262605968","modified":"2024-05-04T12:07:48.391828Z","created":"2024-05-04T12:07:48.391828Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi079","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:3300","nonce":0},{"type":"v1","addr":"172.21.15.79:6789","nonce":0}]},"addr":"172.21.15.79:6789/0","public_addr":"172.21.15.79:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-05-04T12:09:31.423 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:31.423+0000 7f6b772bf700 1 -- 172.21.15.146:0/3297162224 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7f6b5c0383c0 msgr2=0x7f6b5c03a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:09:31.423 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:31.423+0000 7f6b772bf700 1 --2- 172.21.15.146:0/3297162224 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7f6b5c0383c0 0x7f6b5c03a880 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f6b6c006fd0 tx=0x7f6b6c006e40 comp rx=0 tx=0).stop 2024-05-04T12:09:31.423 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:31.423+0000 7f6b772bf700 1 -- 172.21.15.146:0/3297162224 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6b701008a0 msgr2=0x7f6b701959a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:09:31.423 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:31.423+0000 7f6b772bf700 1 --2- 172.21.15.146:0/3297162224 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6b701008a0 0x7f6b701959a0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f6b60005b40 tx=0x7f6b60004740 comp rx=0 tx=0).stop 2024-05-04T12:09:31.423 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:31.424+0000 7f6b772bf700 1 -- 172.21.15.146:0/3297162224 shutdown_connections 2024-05-04T12:09:31.423 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:31.424+0000 7f6b772bf700 1 --2- 172.21.15.146:0/3297162224 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7f6b5c0383c0 0x7f6b5c03a880 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:09:31.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:31.424+0000 7f6b772bf700 1 --2- 172.21.15.146:0/3297162224 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6b701008a0 0x7f6b701959a0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:09:31.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:31.424+0000 7f6b772bf700 1 -- 172.21.15.146:0/3297162224 >> 172.21.15.146:0/3297162224 conn(0x7f6b700fbe20 msgr2=0x7f6b700fcb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:09:31.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:31.424+0000 7f6b772bf700 1 -- 172.21.15.146:0/3297162224 shutdown_connections 2024-05-04T12:09:31.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:31.424+0000 7f6b772bf700 1 -- 172.21.15.146:0/3297162224 wait complete. 2024-05-04T12:09:31.424 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-05-04T12:09:31.457 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:09:31 smithi079 ceph-mon[27222]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-04T12:09:32.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:09:32 smithi079 ceph-mon[27222]: from='client.? 172.21.15.146:0/3297162224' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-04T12:09:33.010 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-05-04T12:09:33.010 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d9b985c4-0a0e-11ef-bc97-c7b262605968 -- ceph mon dump -f json 2024-05-04T12:09:33.211 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /etc/ceph/ceph.conf 2024-05-04T12:09:33.263 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /etc/ceph/ceph.conf 2024-05-04T12:09:33.614 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:09:33 smithi079 ceph-mon[27222]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-04T12:09:33.615 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:09:33 smithi079 ceph-mon[27222]: from='mgr.14164 172.21.15.79:0/395052435' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:09:33.615 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:09:33 smithi079 ceph-mon[27222]: from='mgr.14164 172.21.15.79:0/395052435' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:09:33.615 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:09:33 smithi079 ceph-mon[27222]: from='mgr.14164 172.21.15.79:0/395052435' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:09:33.615 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:09:33 smithi079 ceph-mon[27222]: from='mgr.14164 172.21.15.79:0/395052435' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:09:33.615 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:09:33 smithi079 ceph-mon[27222]: from='mgr.14164 172.21.15.79:0/395052435' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:09:33.615 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:09:33 smithi079 ceph-mon[27222]: from='mgr.14164 172.21.15.79:0/395052435' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:09:33.615 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:09:33 smithi079 ceph-mon[27222]: from='mgr.14164 172.21.15.79:0/395052435' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:09:33.615 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:09:33 smithi079 ceph-mon[27222]: from='mgr.14164 172.21.15.79:0/395052435' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:09:33.615 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:09:33 smithi079 ceph-mon[27222]: from='mgr.14164 172.21.15.79:0/395052435' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-05-04T12:09:33.615 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:09:33 smithi079 ceph-mon[27222]: from='mgr.14164 172.21.15.79:0/395052435' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:09:33.615 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:09:33 smithi079 ceph-mon[27222]: from='mgr.14164 172.21.15.79:0/395052435' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:09:34.695 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:09:34 smithi079 ceph-mon[27222]: Regenerating cephadm self-signed grafana TLS certificates 2024-05-04T12:09:34.695 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:09:34 smithi079 ceph-mon[27222]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-05-04T12:09:34.695 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:09:34 smithi079 ceph-mon[27222]: Deploying daemon grafana.smithi079 on smithi079 2024-05-04T12:09:34.825 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:34.821+0000 7f1db8a64700 1 -- 172.21.15.146:0/2260667135 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1db41013b0 msgr2=0x7f1db41037a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:09:34.825 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:34.821+0000 7f1db8a64700 1 --2- 172.21.15.146:0/2260667135 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1db41013b0 0x7f1db41037a0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f1da4009910 tx=0x7f1da4009c20 comp rx=0 tx=0).stop 2024-05-04T12:09:34.825 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:34.821+0000 7f1db8a64700 1 -- 172.21.15.146:0/2260667135 shutdown_connections 2024-05-04T12:09:34.825 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:34.821+0000 7f1db8a64700 1 --2- 172.21.15.146:0/2260667135 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1db41013b0 0x7f1db41037a0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:09:34.826 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:34.821+0000 7f1db8a64700 1 -- 172.21.15.146:0/2260667135 >> 172.21.15.146:0/2260667135 conn(0x7f1db40fad20 msgr2=0x7f1db40fd160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:09:34.826 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:34.821+0000 7f1db8a64700 1 -- 172.21.15.146:0/2260667135 shutdown_connections 2024-05-04T12:09:34.826 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:34.821+0000 7f1db8a64700 1 -- 172.21.15.146:0/2260667135 wait complete. 2024-05-04T12:09:34.826 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:34.822+0000 7f1db8a64700 1 Processor -- start 2024-05-04T12:09:34.826 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:34.822+0000 7f1db8a64700 1 -- start start 2024-05-04T12:09:34.826 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:34.822+0000 7f1db8a64700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1db41013b0 0x7f1db4195980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:09:34.827 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:34.822+0000 7f1db8a64700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1db4195ec0 con 0x7f1db41013b0 2024-05-04T12:09:34.827 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:34.822+0000 7f1db259c700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1db41013b0 0x7f1db4195980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:09:34.827 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:34.822+0000 7f1db259c700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1db41013b0 0x7f1db4195980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.146:48654/0 (socket says 172.21.15.146:48654) 2024-05-04T12:09:34.827 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:34.822+0000 7f1db259c700 1 -- 172.21.15.146:0/2923464966 learned_addr learned my addr 172.21.15.146:0/2923464966 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-05-04T12:09:34.827 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:34.822+0000 7f1db259c700 1 -- 172.21.15.146:0/2923464966 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1da40095f0 con 0x7f1db41013b0 2024-05-04T12:09:34.827 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:34.822+0000 7f1db259c700 1 --2- 172.21.15.146:0/2923464966 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1db41013b0 0x7f1db4195980 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f1da4005b40 tx=0x7f1da4004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:09:34.828 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:34.823+0000 7f1da37fe700 1 -- 172.21.15.146:0/2923464966 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1da401d070 con 0x7f1db41013b0 2024-05-04T12:09:34.828 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:34.823+0000 7f1da37fe700 1 -- 172.21.15.146:0/2923464966 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f1da400be10 con 0x7f1db41013b0 2024-05-04T12:09:34.828 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:34.823+0000 7f1db8a64700 1 -- 172.21.15.146:0/2923464966 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1db41960c0 con 0x7f1db41013b0 2024-05-04T12:09:34.828 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:34.823+0000 7f1da37fe700 1 -- 172.21.15.146:0/2923464966 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1da400f460 con 0x7f1db41013b0 2024-05-04T12:09:34.828 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:34.823+0000 7f1db8a64700 1 -- 172.21.15.146:0/2923464966 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1db4196560 con 0x7f1db41013b0 2024-05-04T12:09:34.829 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:34.823+0000 7f1da37fe700 1 -- 172.21.15.146:0/2923464966 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 13) v1 ==== 45339+0+0 (secure 0 0 0) 0x7f1da400b960 con 0x7f1db41013b0 2024-05-04T12:09:34.829 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:34.823+0000 7f1db8a64700 1 -- 172.21.15.146:0/2923464966 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1db418f4a0 con 0x7f1db41013b0 2024-05-04T12:09:34.829 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:34.824+0000 7f1da37fe700 1 --2- 172.21.15.146:0/2923464966 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7f1d9c0383c0 0x7f1d9c03a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:09:34.829 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:34.824+0000 7f1da37fe700 1 -- 172.21.15.146:0/2923464966 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f1da404c4b0 con 0x7f1db41013b0 2024-05-04T12:09:34.829 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:34.824+0000 7f1db1d9b700 1 --2- 172.21.15.146:0/2923464966 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7f1d9c0383c0 0x7f1d9c03a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:09:34.829 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:34.824+0000 7f1db1d9b700 1 --2- 172.21.15.146:0/2923464966 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7f1d9c0383c0 0x7f1d9c03a880 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f1da8006fd0 tx=0x7f1da8006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:09:34.830 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:34.830+0000 7f1da37fe700 1 -- 172.21.15.146:0/2923464966 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f1da402a360 con 0x7f1db41013b0 2024-05-04T12:09:35.092 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:35.092+0000 7f1db8a64700 1 -- 172.21.15.146:0/2923464966 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1db402cc70 con 0x7f1db41013b0 2024-05-04T12:09:35.092 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:35.093+0000 7f1da37fe700 1 -- 172.21.15.146:0/2923464966 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f1da4027030 con 0x7f1db41013b0 2024-05-04T12:09:35.092 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-05-04T12:09:35.092 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"d9b985c4-0a0e-11ef-bc97-c7b262605968","modified":"2024-05-04T12:07:48.391828Z","created":"2024-05-04T12:07:48.391828Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi079","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:3300","nonce":0},{"type":"v1","addr":"172.21.15.79:6789","nonce":0}]},"addr":"172.21.15.79:6789/0","public_addr":"172.21.15.79:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-05-04T12:09:35.095 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:35.095+0000 7f1db8a64700 1 -- 172.21.15.146:0/2923464966 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7f1d9c0383c0 msgr2=0x7f1d9c03a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:09:35.095 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:35.095+0000 7f1db8a64700 1 --2- 172.21.15.146:0/2923464966 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7f1d9c0383c0 0x7f1d9c03a880 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f1da8006fd0 tx=0x7f1da8006e40 comp rx=0 tx=0).stop 2024-05-04T12:09:35.095 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:35.095+0000 7f1db8a64700 1 -- 172.21.15.146:0/2923464966 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1db41013b0 msgr2=0x7f1db4195980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:09:35.095 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:35.095+0000 7f1db8a64700 1 --2- 172.21.15.146:0/2923464966 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1db41013b0 0x7f1db4195980 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f1da4005b40 tx=0x7f1da4004740 comp rx=0 tx=0).stop 2024-05-04T12:09:35.095 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:35.095+0000 7f1db8a64700 1 -- 172.21.15.146:0/2923464966 shutdown_connections 2024-05-04T12:09:35.095 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:35.095+0000 7f1db8a64700 1 --2- 172.21.15.146:0/2923464966 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7f1d9c0383c0 0x7f1d9c03a880 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:09:35.096 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:35.095+0000 7f1db8a64700 1 --2- 172.21.15.146:0/2923464966 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1db41013b0 0x7f1db4195980 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:09:35.096 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:35.095+0000 7f1db8a64700 1 -- 172.21.15.146:0/2923464966 >> 172.21.15.146:0/2923464966 conn(0x7f1db40fad20 msgr2=0x7f1db40fb9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:09:35.096 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:35.096+0000 7f1db8a64700 1 -- 172.21.15.146:0/2923464966 shutdown_connections 2024-05-04T12:09:35.096 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:35.096+0000 7f1db8a64700 1 -- 172.21.15.146:0/2923464966 wait complete. 2024-05-04T12:09:35.096 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-05-04T12:09:35.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:09:35 smithi079 ceph-mon[27222]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-04T12:09:35.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:09:35 smithi079 ceph-mon[27222]: from='client.? 172.21.15.146:0/2923464966' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-04T12:09:36.670 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-05-04T12:09:36.671 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d9b985c4-0a0e-11ef-bc97-c7b262605968 -- ceph mon dump -f json 2024-05-04T12:09:36.873 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /etc/ceph/ceph.conf 2024-05-04T12:09:36.919 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /etc/ceph/ceph.conf 2024-05-04T12:09:37.444 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:09:36 smithi079 ceph-mon[27222]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-04T12:09:38.343 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:38.343+0000 7f338030a700 1 -- 172.21.15.146:0/1169028439 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3378101310 msgr2=0x7f3378103700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:09:38.343 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:38.343+0000 7f338030a700 1 --2- 172.21.15.146:0/1169028439 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3378101310 0x7f3378103700 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f3374009910 tx=0x7f3374009c20 comp rx=0 tx=0).stop 2024-05-04T12:09:38.343 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:38.343+0000 7f338030a700 1 -- 172.21.15.146:0/1169028439 shutdown_connections 2024-05-04T12:09:38.343 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:38.343+0000 7f338030a700 1 --2- 172.21.15.146:0/1169028439 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3378101310 0x7f3378103700 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:09:38.343 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:38.343+0000 7f338030a700 1 -- 172.21.15.146:0/1169028439 >> 172.21.15.146:0/1169028439 conn(0x7f33780fad20 msgr2=0x7f33780fd160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:09:38.343 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:38.343+0000 7f338030a700 1 -- 172.21.15.146:0/1169028439 shutdown_connections 2024-05-04T12:09:38.343 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:38.343+0000 7f338030a700 1 -- 172.21.15.146:0/1169028439 wait complete. 2024-05-04T12:09:38.343 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:38.344+0000 7f338030a700 1 Processor -- start 2024-05-04T12:09:38.344 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:38.344+0000 7f338030a700 1 -- start start 2024-05-04T12:09:38.344 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:38.344+0000 7f338030a700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3378101310 0x7f33781958f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:09:38.344 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:38.344+0000 7f338030a700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3378195e30 con 0x7f3378101310 2024-05-04T12:09:38.344 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:38.344+0000 7f337e0a6700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3378101310 0x7f33781958f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:09:38.344 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:38.344+0000 7f337e0a6700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3378101310 0x7f33781958f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.146:48682/0 (socket says 172.21.15.146:48682) 2024-05-04T12:09:38.344 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:38.344+0000 7f337e0a6700 1 -- 172.21.15.146:0/3805598327 learned_addr learned my addr 172.21.15.146:0/3805598327 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-05-04T12:09:38.344 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:38.344+0000 7f337e0a6700 1 -- 172.21.15.146:0/3805598327 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33740095f0 con 0x7f3378101310 2024-05-04T12:09:38.344 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:38.344+0000 7f337e0a6700 1 --2- 172.21.15.146:0/3805598327 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3378101310 0x7f33781958f0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f33740092e0 tx=0x7f3374009e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:09:38.344 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:38.345+0000 7f336f7fe700 1 -- 172.21.15.146:0/3805598327 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f337401d070 con 0x7f3378101310 2024-05-04T12:09:38.344 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:38.345+0000 7f336f7fe700 1 -- 172.21.15.146:0/3805598327 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f337400be10 con 0x7f3378101310 2024-05-04T12:09:38.344 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:38.345+0000 7f338030a700 1 -- 172.21.15.146:0/3805598327 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3378196030 con 0x7f3378101310 2024-05-04T12:09:38.344 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:38.345+0000 7f336f7fe700 1 -- 172.21.15.146:0/3805598327 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f337400f460 con 0x7f3378101310 2024-05-04T12:09:38.344 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:38.345+0000 7f338030a700 1 -- 172.21.15.146:0/3805598327 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33781964d0 con 0x7f3378101310 2024-05-04T12:09:38.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:38.346+0000 7f338030a700 1 -- 172.21.15.146:0/3805598327 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f337818f4a0 con 0x7f3378101310 2024-05-04T12:09:38.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:38.346+0000 7f336f7fe700 1 -- 172.21.15.146:0/3805598327 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 13) v1 ==== 45339+0+0 (secure 0 0 0) 0x7f3374022470 con 0x7f3378101310 2024-05-04T12:09:38.346 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:38.346+0000 7f336f7fe700 1 --2- 172.21.15.146:0/3805598327 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7f33640383c0 0x7f336403a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:09:38.346 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:38.346+0000 7f336f7fe700 1 -- 172.21.15.146:0/3805598327 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f337404c620 con 0x7f3378101310 2024-05-04T12:09:38.346 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:38.346+0000 7f337d8a5700 1 --2- 172.21.15.146:0/3805598327 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7f33640383c0 0x7f336403a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:09:38.346 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:38.346+0000 7f337d8a5700 1 --2- 172.21.15.146:0/3805598327 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7f33640383c0 0x7f336403a880 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f3368006fd0 tx=0x7f3368006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:09:38.349 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:38.351+0000 7f336f7fe700 1 -- 172.21.15.146:0/3805598327 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f337400b960 con 0x7f3378101310 2024-05-04T12:09:38.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:38.617+0000 7f338030a700 1 -- 172.21.15.146:0/3805598327 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f337802cc70 con 0x7f3378101310 2024-05-04T12:09:38.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:38.618+0000 7f336f7fe700 1 -- 172.21.15.146:0/3805598327 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f3374027020 con 0x7f3378101310 2024-05-04T12:09:38.617 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-05-04T12:09:38.617 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"d9b985c4-0a0e-11ef-bc97-c7b262605968","modified":"2024-05-04T12:07:48.391828Z","created":"2024-05-04T12:07:48.391828Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi079","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:3300","nonce":0},{"type":"v1","addr":"172.21.15.79:6789","nonce":0}]},"addr":"172.21.15.79:6789/0","public_addr":"172.21.15.79:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-05-04T12:09:38.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:38.620+0000 7f338030a700 1 -- 172.21.15.146:0/3805598327 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7f33640383c0 msgr2=0x7f336403a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:09:38.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:38.620+0000 7f338030a700 1 --2- 172.21.15.146:0/3805598327 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7f33640383c0 0x7f336403a880 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f3368006fd0 tx=0x7f3368006e40 comp rx=0 tx=0).stop 2024-05-04T12:09:38.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:38.620+0000 7f338030a700 1 -- 172.21.15.146:0/3805598327 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3378101310 msgr2=0x7f33781958f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:09:38.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:38.620+0000 7f338030a700 1 --2- 172.21.15.146:0/3805598327 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3378101310 0x7f33781958f0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f33740092e0 tx=0x7f3374009e60 comp rx=0 tx=0).stop 2024-05-04T12:09:38.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:38.620+0000 7f338030a700 1 -- 172.21.15.146:0/3805598327 shutdown_connections 2024-05-04T12:09:38.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:38.620+0000 7f338030a700 1 --2- 172.21.15.146:0/3805598327 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7f33640383c0 0x7f336403a880 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:09:38.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:38.620+0000 7f338030a700 1 --2- 172.21.15.146:0/3805598327 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3378101310 0x7f33781958f0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:09:38.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:38.620+0000 7f338030a700 1 -- 172.21.15.146:0/3805598327 >> 172.21.15.146:0/3805598327 conn(0x7f33780fad20 msgr2=0x7f33780fb9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:09:38.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:38.620+0000 7f338030a700 1 -- 172.21.15.146:0/3805598327 shutdown_connections 2024-05-04T12:09:38.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:38.620+0000 7f338030a700 1 -- 172.21.15.146:0/3805598327 wait complete. 2024-05-04T12:09:38.620 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-05-04T12:09:39.694 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:09:39 smithi079 ceph-mon[27222]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-04T12:09:39.695 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:09:39 smithi079 ceph-mon[27222]: from='client.? 172.21.15.146:0/3805598327' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-04T12:09:40.190 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-05-04T12:09:40.191 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d9b985c4-0a0e-11ef-bc97-c7b262605968 -- ceph mon dump -f json 2024-05-04T12:09:40.380 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /etc/ceph/ceph.conf 2024-05-04T12:09:40.427 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /etc/ceph/ceph.conf 2024-05-04T12:09:41.695 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:09:41 smithi079 ceph-mon[27222]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-04T12:09:41.797 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:41.797+0000 7efccf594700 1 -- 172.21.15.146:0/1522896781 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efcc81008a0 msgr2=0x7efcc8100cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:09:41.797 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:41.797+0000 7efccf594700 1 --2- 172.21.15.146:0/1522896781 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efcc81008a0 0x7efcc8100cc0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7efcc4009910 tx=0x7efcc4009c20 comp rx=0 tx=0).stop 2024-05-04T12:09:41.797 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:41.797+0000 7efccf594700 1 -- 172.21.15.146:0/1522896781 shutdown_connections 2024-05-04T12:09:41.797 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:41.797+0000 7efccf594700 1 --2- 172.21.15.146:0/1522896781 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efcc81008a0 0x7efcc8100cc0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:09:41.797 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:41.797+0000 7efccf594700 1 -- 172.21.15.146:0/1522896781 >> 172.21.15.146:0/1522896781 conn(0x7efcc80fbe20 msgr2=0x7efcc80fe280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:09:41.797 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:41.797+0000 7efccf594700 1 -- 172.21.15.146:0/1522896781 shutdown_connections 2024-05-04T12:09:41.798 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:41.797+0000 7efccf594700 1 -- 172.21.15.146:0/1522896781 wait complete. 2024-05-04T12:09:41.798 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:41.797+0000 7efccf594700 1 Processor -- start 2024-05-04T12:09:41.798 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:41.797+0000 7efccf594700 1 -- start start 2024-05-04T12:09:41.798 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:41.798+0000 7efccf594700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efcc81008a0 0x7efcc8074b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:09:41.798 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:41.798+0000 7efccf594700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcc8075070 con 0x7efcc81008a0 2024-05-04T12:09:41.798 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:41.798+0000 7efccd330700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efcc81008a0 0x7efcc8074b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:09:41.798 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:41.798+0000 7efccd330700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efcc81008a0 0x7efcc8074b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.146:48704/0 (socket says 172.21.15.146:48704) 2024-05-04T12:09:41.798 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:41.798+0000 7efccd330700 1 -- 172.21.15.146:0/3362154983 learned_addr learned my addr 172.21.15.146:0/3362154983 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-05-04T12:09:41.798 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:41.798+0000 7efccd330700 1 -- 172.21.15.146:0/3362154983 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcc40095f0 con 0x7efcc81008a0 2024-05-04T12:09:41.798 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:41.798+0000 7efccd330700 1 --2- 172.21.15.146:0/3362154983 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efcc81008a0 0x7efcc8074b30 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7efcc4005b40 tx=0x7efcc4004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:09:41.798 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:41.798+0000 7efcbe7fc700 1 -- 172.21.15.146:0/3362154983 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efcc401d070 con 0x7efcc81008a0 2024-05-04T12:09:41.798 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:41.798+0000 7efccf594700 1 -- 172.21.15.146:0/3362154983 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcc8073180 con 0x7efcc81008a0 2024-05-04T12:09:41.798 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:41.798+0000 7efcbe7fc700 1 -- 172.21.15.146:0/3362154983 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7efcc400be10 con 0x7efcc81008a0 2024-05-04T12:09:41.799 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:41.798+0000 7efcbe7fc700 1 -- 172.21.15.146:0/3362154983 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efcc400f460 con 0x7efcc81008a0 2024-05-04T12:09:41.799 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:41.798+0000 7efccf594700 1 -- 172.21.15.146:0/3362154983 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcc8073620 con 0x7efcc81008a0 2024-05-04T12:09:41.801 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:41.799+0000 7efcbe7fc700 1 -- 172.21.15.146:0/3362154983 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 13) v1 ==== 45339+0+0 (secure 0 0 0) 0x7efcc400f6d0 con 0x7efcc81008a0 2024-05-04T12:09:41.801 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:41.799+0000 7efcbe7fc700 1 --2- 172.21.15.146:0/3362154983 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7efcb40383c0 0x7efcb403a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:09:41.802 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:41.799+0000 7efcbe7fc700 1 -- 172.21.15.146:0/3362154983 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7efcc404c4c0 con 0x7efcc81008a0 2024-05-04T12:09:41.802 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:41.799+0000 7efccf594700 1 -- 172.21.15.146:0/3362154983 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcc80623c0 con 0x7efcc81008a0 2024-05-04T12:09:41.802 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:41.800+0000 7efcccb2f700 1 --2- 172.21.15.146:0/3362154983 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7efcb40383c0 0x7efcb403a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:09:41.802 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:41.800+0000 7efcccb2f700 1 --2- 172.21.15.146:0/3362154983 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7efcb40383c0 0x7efcb403a880 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7efcb8006fd0 tx=0x7efcb8006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:09:41.802 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:41.805+0000 7efcbe7fc700 1 -- 172.21.15.146:0/3362154983 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7efcc4027070 con 0x7efcc81008a0 2024-05-04T12:09:42.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:42.065+0000 7efccf594700 1 -- 172.21.15.146:0/3362154983 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7efcc819b9f0 con 0x7efcc81008a0 2024-05-04T12:09:42.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:42.066+0000 7efcbe7fc700 1 -- 172.21.15.146:0/3362154983 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7efcc402c440 con 0x7efcc81008a0 2024-05-04T12:09:42.065 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-05-04T12:09:42.065 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"d9b985c4-0a0e-11ef-bc97-c7b262605968","modified":"2024-05-04T12:07:48.391828Z","created":"2024-05-04T12:07:48.391828Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi079","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:3300","nonce":0},{"type":"v1","addr":"172.21.15.79:6789","nonce":0}]},"addr":"172.21.15.79:6789/0","public_addr":"172.21.15.79:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-05-04T12:09:42.067 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:42.068+0000 7efccf594700 1 -- 172.21.15.146:0/3362154983 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7efcb40383c0 msgr2=0x7efcb403a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:09:42.067 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:42.068+0000 7efccf594700 1 --2- 172.21.15.146:0/3362154983 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7efcb40383c0 0x7efcb403a880 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7efcb8006fd0 tx=0x7efcb8006e40 comp rx=0 tx=0).stop 2024-05-04T12:09:42.067 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:42.068+0000 7efccf594700 1 -- 172.21.15.146:0/3362154983 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efcc81008a0 msgr2=0x7efcc8074b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:09:42.067 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:42.068+0000 7efccf594700 1 --2- 172.21.15.146:0/3362154983 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efcc81008a0 0x7efcc8074b30 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7efcc4005b40 tx=0x7efcc4004740 comp rx=0 tx=0).stop 2024-05-04T12:09:42.067 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:42.068+0000 7efccf594700 1 -- 172.21.15.146:0/3362154983 shutdown_connections 2024-05-04T12:09:42.067 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:42.068+0000 7efccf594700 1 --2- 172.21.15.146:0/3362154983 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7efcb40383c0 0x7efcb403a880 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:09:42.067 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:42.068+0000 7efccf594700 1 --2- 172.21.15.146:0/3362154983 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efcc81008a0 0x7efcc8074b30 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:09:42.067 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:42.068+0000 7efccf594700 1 -- 172.21.15.146:0/3362154983 >> 172.21.15.146:0/3362154983 conn(0x7efcc80fbe20 msgr2=0x7efcc80fcb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:09:42.067 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:42.069+0000 7efccf594700 1 -- 172.21.15.146:0/3362154983 shutdown_connections 2024-05-04T12:09:42.068 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:42.069+0000 7efccf594700 1 -- 172.21.15.146:0/3362154983 wait complete. 2024-05-04T12:09:42.068 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-05-04T12:09:42.695 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:09:42 smithi079 ceph-mon[27222]: from='client.? 172.21.15.146:0/3362154983' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-04T12:09:43.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:09:43 smithi079 ceph-mon[27222]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-04T12:09:43.594 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-05-04T12:09:43.594 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d9b985c4-0a0e-11ef-bc97-c7b262605968 -- ceph mon dump -f json 2024-05-04T12:09:43.795 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /etc/ceph/ceph.conf 2024-05-04T12:09:43.842 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /etc/ceph/ceph.conf 2024-05-04T12:09:45.223 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:45.224+0000 7f7293aa6700 1 -- 172.21.15.146:0/2073700121 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f728c1008a0 msgr2=0x7f728c100cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:09:45.223 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:45.224+0000 7f7293aa6700 1 --2- 172.21.15.146:0/2073700121 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f728c1008a0 0x7f728c100cc0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f727c009910 tx=0x7f727c009c20 comp rx=0 tx=0).stop 2024-05-04T12:09:45.223 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:45.224+0000 7f7293aa6700 1 -- 172.21.15.146:0/2073700121 shutdown_connections 2024-05-04T12:09:45.223 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:45.224+0000 7f7293aa6700 1 --2- 172.21.15.146:0/2073700121 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f728c1008a0 0x7f728c100cc0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:09:45.224 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:45.224+0000 7f7293aa6700 1 -- 172.21.15.146:0/2073700121 >> 172.21.15.146:0/2073700121 conn(0x7f728c0fbe20 msgr2=0x7f728c0fe280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:09:45.224 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:45.224+0000 7f7293aa6700 1 -- 172.21.15.146:0/2073700121 shutdown_connections 2024-05-04T12:09:45.224 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:45.224+0000 7f7293aa6700 1 -- 172.21.15.146:0/2073700121 wait complete. 2024-05-04T12:09:45.224 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:45.224+0000 7f7293aa6700 1 Processor -- start 2024-05-04T12:09:45.224 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:45.225+0000 7f7293aa6700 1 -- start start 2024-05-04T12:09:45.224 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:45.225+0000 7f7293aa6700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f728c1008a0 0x7f728c074b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:09:45.224 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:45.225+0000 7f7293aa6700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f728c075070 con 0x7f728c1008a0 2024-05-04T12:09:45.224 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:45.225+0000 7f7291842700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f728c1008a0 0x7f728c074b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:09:45.224 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:45.225+0000 7f7291842700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f728c1008a0 0x7f728c074b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.146:39910/0 (socket says 172.21.15.146:39910) 2024-05-04T12:09:45.224 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:45.225+0000 7f7291842700 1 -- 172.21.15.146:0/489002904 learned_addr learned my addr 172.21.15.146:0/489002904 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-05-04T12:09:45.224 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:45.225+0000 7f7291842700 1 -- 172.21.15.146:0/489002904 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f727c0095f0 con 0x7f728c1008a0 2024-05-04T12:09:45.226 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:45.225+0000 7f7291842700 1 --2- 172.21.15.146:0/489002904 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f728c1008a0 0x7f728c074b30 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f727c005950 tx=0x7f727c009e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:09:45.226 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:45.226+0000 7f7282ffd700 1 -- 172.21.15.146:0/489002904 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f727c01d070 con 0x7f728c1008a0 2024-05-04T12:09:45.226 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:45.226+0000 7f7293aa6700 1 -- 172.21.15.146:0/489002904 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f728c073180 con 0x7f728c1008a0 2024-05-04T12:09:45.226 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:45.226+0000 7f7282ffd700 1 -- 172.21.15.146:0/489002904 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f727c00be10 con 0x7f728c1008a0 2024-05-04T12:09:45.226 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:45.226+0000 7f7293aa6700 1 -- 172.21.15.146:0/489002904 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f728c073620 con 0x7f728c1008a0 2024-05-04T12:09:45.226 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:45.226+0000 7f7282ffd700 1 -- 172.21.15.146:0/489002904 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f727c00f460 con 0x7f728c1008a0 2024-05-04T12:09:45.226 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:45.226+0000 7f7293aa6700 1 -- 172.21.15.146:0/489002904 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f728c0623c0 con 0x7f728c1008a0 2024-05-04T12:09:45.226 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:45.227+0000 7f7282ffd700 1 -- 172.21.15.146:0/489002904 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 13) v1 ==== 45339+0+0 (secure 0 0 0) 0x7f727c0227c0 con 0x7f728c1008a0 2024-05-04T12:09:45.226 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:45.227+0000 7f7282ffd700 1 --2- 172.21.15.146:0/489002904 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7f72780383c0 0x7f727803a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:09:45.226 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:45.227+0000 7f7282ffd700 1 -- 172.21.15.146:0/489002904 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f727c04c3c0 con 0x7f728c1008a0 2024-05-04T12:09:45.226 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:45.227+0000 7f7291041700 1 --2- 172.21.15.146:0/489002904 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7f72780383c0 0x7f727803a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:09:45.227 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:45.227+0000 7f7291041700 1 --2- 172.21.15.146:0/489002904 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7f72780383c0 0x7f727803a880 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f7288006fd0 tx=0x7f7288006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:09:45.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:45.232+0000 7f7282ffd700 1 -- 172.21.15.146:0/489002904 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f727c00b960 con 0x7f728c1008a0 2024-05-04T12:09:45.463 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:09:45 smithi079 ceph-mon[27222]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-04T12:09:45.495 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:45.496+0000 7f7293aa6700 1 -- 172.21.15.146:0/489002904 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f728c18eea0 con 0x7f728c1008a0 2024-05-04T12:09:45.495 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:45.497+0000 7f7282ffd700 1 -- 172.21.15.146:0/489002904 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f727c027070 con 0x7f728c1008a0 2024-05-04T12:09:45.496 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-05-04T12:09:45.496 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"d9b985c4-0a0e-11ef-bc97-c7b262605968","modified":"2024-05-04T12:07:48.391828Z","created":"2024-05-04T12:07:48.391828Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi079","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:3300","nonce":0},{"type":"v1","addr":"172.21.15.79:6789","nonce":0}]},"addr":"172.21.15.79:6789/0","public_addr":"172.21.15.79:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-05-04T12:09:45.498 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:45.499+0000 7f7293aa6700 1 -- 172.21.15.146:0/489002904 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7f72780383c0 msgr2=0x7f727803a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:09:45.498 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:45.499+0000 7f7293aa6700 1 --2- 172.21.15.146:0/489002904 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7f72780383c0 0x7f727803a880 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f7288006fd0 tx=0x7f7288006e40 comp rx=0 tx=0).stop 2024-05-04T12:09:45.498 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:45.499+0000 7f7293aa6700 1 -- 172.21.15.146:0/489002904 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f728c1008a0 msgr2=0x7f728c074b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:09:45.498 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:45.499+0000 7f7293aa6700 1 --2- 172.21.15.146:0/489002904 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f728c1008a0 0x7f728c074b30 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f727c005950 tx=0x7f727c009e60 comp rx=0 tx=0).stop 2024-05-04T12:09:45.498 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:45.499+0000 7f7293aa6700 1 -- 172.21.15.146:0/489002904 shutdown_connections 2024-05-04T12:09:45.498 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:45.500+0000 7f7293aa6700 1 --2- 172.21.15.146:0/489002904 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7f72780383c0 0x7f727803a880 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:09:45.498 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:45.500+0000 7f7293aa6700 1 --2- 172.21.15.146:0/489002904 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f728c1008a0 0x7f728c074b30 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:09:45.499 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:45.500+0000 7f7293aa6700 1 -- 172.21.15.146:0/489002904 >> 172.21.15.146:0/489002904 conn(0x7f728c0fbe20 msgr2=0x7f728c0fcb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:09:45.499 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:45.500+0000 7f7293aa6700 1 -- 172.21.15.146:0/489002904 shutdown_connections 2024-05-04T12:09:45.499 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:45.500+0000 7f7293aa6700 1 -- 172.21.15.146:0/489002904 wait complete. 2024-05-04T12:09:45.499 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-05-04T12:09:46.434 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:09:46 smithi079 ceph-mon[27222]: from='client.? 172.21.15.146:0/489002904' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-04T12:09:47.014 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-05-04T12:09:47.014 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d9b985c4-0a0e-11ef-bc97-c7b262605968 -- ceph mon dump -f json 2024-05-04T12:09:47.213 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /etc/ceph/ceph.conf 2024-05-04T12:09:47.260 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /etc/ceph/ceph.conf 2024-05-04T12:09:47.854 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:09:47 smithi079 ceph-mon[27222]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-04T12:09:47.854 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:09:47 smithi079 ceph-mon[27222]: from='mgr.14164 172.21.15.79:0/395052435' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:09:47.854 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:09:47 smithi079 ceph-mon[27222]: from='mgr.14164 172.21.15.79:0/395052435' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:09:47.854 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:09:47 smithi079 ceph-mon[27222]: from='mgr.14164 172.21.15.79:0/395052435' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:09:47.854 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:09:47 smithi079 ceph-mon[27222]: from='mgr.14164 172.21.15.79:0/395052435' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:09:47.854 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:09:47 smithi079 ceph-mon[27222]: from='mgr.14164 172.21.15.79:0/395052435' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:09:47.854 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:09:47 smithi079 ceph-mon[27222]: from='mgr.14164 172.21.15.79:0/395052435' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:09:47.855 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:09:47 smithi079 ceph-mon[27222]: from='mgr.14164 172.21.15.79:0/395052435' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:09:47.855 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:09:47 smithi079 ceph-mon[27222]: from='mgr.14164 172.21.15.79:0/395052435' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:09:48.650 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:48.650+0000 7f8bd2531700 1 -- 172.21.15.146:0/3884089022 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8bcc1008a0 msgr2=0x7f8bcc100cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:09:48.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:48.650+0000 7f8bd2531700 1 --2- 172.21.15.146:0/3884089022 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8bcc1008a0 0x7f8bcc100cc0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f8bc0009910 tx=0x7f8bc0009c20 comp rx=0 tx=0).stop 2024-05-04T12:09:48.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:48.650+0000 7f8bd2531700 1 -- 172.21.15.146:0/3884089022 shutdown_connections 2024-05-04T12:09:48.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:48.650+0000 7f8bd2531700 1 --2- 172.21.15.146:0/3884089022 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8bcc1008a0 0x7f8bcc100cc0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:09:48.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:48.650+0000 7f8bd2531700 1 -- 172.21.15.146:0/3884089022 >> 172.21.15.146:0/3884089022 conn(0x7f8bcc0fbe20 msgr2=0x7f8bcc0fe280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:09:48.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:48.650+0000 7f8bd2531700 1 -- 172.21.15.146:0/3884089022 shutdown_connections 2024-05-04T12:09:48.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:48.650+0000 7f8bd2531700 1 -- 172.21.15.146:0/3884089022 wait complete. 2024-05-04T12:09:48.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:48.651+0000 7f8bd2531700 1 Processor -- start 2024-05-04T12:09:48.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:48.651+0000 7f8bd2531700 1 -- start start 2024-05-04T12:09:48.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:48.651+0000 7f8bd2531700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8bcc1008a0 0x7f8bcc1959f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:09:48.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:48.651+0000 7f8bd2531700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bcc195f30 con 0x7f8bcc1008a0 2024-05-04T12:09:48.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:48.651+0000 7f8bcbfff700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8bcc1008a0 0x7f8bcc1959f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:09:48.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:48.651+0000 7f8bcbfff700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8bcc1008a0 0x7f8bcc1959f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.146:39938/0 (socket says 172.21.15.146:39938) 2024-05-04T12:09:48.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:48.651+0000 7f8bcbfff700 1 -- 172.21.15.146:0/1220676242 learned_addr learned my addr 172.21.15.146:0/1220676242 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-05-04T12:09:48.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:48.651+0000 7f8bcbfff700 1 -- 172.21.15.146:0/1220676242 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bc00095f0 con 0x7f8bcc1008a0 2024-05-04T12:09:48.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:48.652+0000 7f8bcbfff700 1 --2- 172.21.15.146:0/1220676242 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8bcc1008a0 0x7f8bcc1959f0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f8bc0005b40 tx=0x7f8bc0004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:09:48.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:48.652+0000 7f8bc97fa700 1 -- 172.21.15.146:0/1220676242 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8bc001d070 con 0x7f8bcc1008a0 2024-05-04T12:09:48.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:48.652+0000 7f8bd2531700 1 -- 172.21.15.146:0/1220676242 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bcc196130 con 0x7f8bcc1008a0 2024-05-04T12:09:48.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:48.652+0000 7f8bc97fa700 1 -- 172.21.15.146:0/1220676242 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f8bc000be10 con 0x7f8bcc1008a0 2024-05-04T12:09:48.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:48.652+0000 7f8bc97fa700 1 -- 172.21.15.146:0/1220676242 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8bc000f460 con 0x7f8bcc1008a0 2024-05-04T12:09:48.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:48.652+0000 7f8bd2531700 1 -- 172.21.15.146:0/1220676242 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bcc1965d0 con 0x7f8bcc1008a0 2024-05-04T12:09:48.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:48.653+0000 7f8bc97fa700 1 -- 172.21.15.146:0/1220676242 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 13) v1 ==== 45339+0+0 (secure 0 0 0) 0x7f8bc00227e0 con 0x7f8bcc1008a0 2024-05-04T12:09:48.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:48.653+0000 7f8bd2531700 1 -- 172.21.15.146:0/1220676242 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8bcc18f4c0 con 0x7f8bcc1008a0 2024-05-04T12:09:48.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:48.653+0000 7f8bc97fa700 1 --2- 172.21.15.146:0/1220676242 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7f8bb40383c0 0x7f8bb403a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:09:48.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:48.653+0000 7f8bc97fa700 1 -- 172.21.15.146:0/1220676242 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f8bc004c560 con 0x7f8bcc1008a0 2024-05-04T12:09:48.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:48.653+0000 7f8bcb7fe700 1 --2- 172.21.15.146:0/1220676242 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7f8bb40383c0 0x7f8bb403a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:09:48.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:48.654+0000 7f8bcb7fe700 1 --2- 172.21.15.146:0/1220676242 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7f8bb40383c0 0x7f8bb403a880 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f8bbc006fd0 tx=0x7f8bbc006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:09:48.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:48.663+0000 7f8bc97fa700 1 -- 172.21.15.146:0/1220676242 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f8bc002bba0 con 0x7f8bcc1008a0 2024-05-04T12:09:48.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:48.926+0000 7f8bd2531700 1 -- 172.21.15.146:0/1220676242 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8bcc02cc70 con 0x7f8bcc1008a0 2024-05-04T12:09:48.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:48.927+0000 7f8bc97fa700 1 -- 172.21.15.146:0/1220676242 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f8bc0027020 con 0x7f8bcc1008a0 2024-05-04T12:09:48.925 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-05-04T12:09:48.925 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"d9b985c4-0a0e-11ef-bc97-c7b262605968","modified":"2024-05-04T12:07:48.391828Z","created":"2024-05-04T12:07:48.391828Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi079","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:3300","nonce":0},{"type":"v1","addr":"172.21.15.79:6789","nonce":0}]},"addr":"172.21.15.79:6789/0","public_addr":"172.21.15.79:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-05-04T12:09:48.928 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:48.929+0000 7f8bd2531700 1 -- 172.21.15.146:0/1220676242 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7f8bb40383c0 msgr2=0x7f8bb403a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:09:48.928 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:48.929+0000 7f8bd2531700 1 --2- 172.21.15.146:0/1220676242 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7f8bb40383c0 0x7f8bb403a880 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f8bbc006fd0 tx=0x7f8bbc006e40 comp rx=0 tx=0).stop 2024-05-04T12:09:48.928 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:48.929+0000 7f8bd2531700 1 -- 172.21.15.146:0/1220676242 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8bcc1008a0 msgr2=0x7f8bcc1959f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:09:48.928 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:48.929+0000 7f8bd2531700 1 --2- 172.21.15.146:0/1220676242 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8bcc1008a0 0x7f8bcc1959f0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f8bc0005b40 tx=0x7f8bc0004740 comp rx=0 tx=0).stop 2024-05-04T12:09:48.928 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:48.929+0000 7f8bd2531700 1 -- 172.21.15.146:0/1220676242 shutdown_connections 2024-05-04T12:09:48.928 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:48.929+0000 7f8bd2531700 1 --2- 172.21.15.146:0/1220676242 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7f8bb40383c0 0x7f8bb403a880 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:09:48.928 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:48.929+0000 7f8bd2531700 1 --2- 172.21.15.146:0/1220676242 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8bcc1008a0 0x7f8bcc1959f0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:09:48.928 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:48.929+0000 7f8bd2531700 1 -- 172.21.15.146:0/1220676242 >> 172.21.15.146:0/1220676242 conn(0x7f8bcc0fbe20 msgr2=0x7f8bcc0fcb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:09:48.929 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:48.930+0000 7f8bd2531700 1 -- 172.21.15.146:0/1220676242 shutdown_connections 2024-05-04T12:09:48.929 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:48.930+0000 7f8bd2531700 1 -- 172.21.15.146:0/1220676242 wait complete. 2024-05-04T12:09:48.929 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-05-04T12:09:48.945 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:09:48 smithi079 ceph-mon[27222]: Deploying daemon prometheus.smithi079 on smithi079 2024-05-04T12:09:48.945 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:09:48 smithi079 ceph-mon[27222]: from='mgr.14164 172.21.15.79:0/395052435' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:09:49.945 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:09:49 smithi079 ceph-mon[27222]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-04T12:09:49.945 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:09:49 smithi079 ceph-mon[27222]: from='client.? 172.21.15.146:0/1220676242' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-04T12:09:50.591 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-05-04T12:09:50.592 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d9b985c4-0a0e-11ef-bc97-c7b262605968 -- ceph mon dump -f json 2024-05-04T12:09:50.817 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /etc/ceph/ceph.conf 2024-05-04T12:09:50.862 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /etc/ceph/ceph.conf 2024-05-04T12:09:51.944 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:09:51 smithi079 ceph-mon[27222]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-04T12:09:52.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:52.270+0000 7fac18e3e700 1 -- 172.21.15.146:0/1460178992 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fac141008a0 msgr2=0x7fac14100cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:09:52.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:52.270+0000 7fac18e3e700 1 --2- 172.21.15.146:0/1460178992 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fac141008a0 0x7fac14100cc0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fac04009910 tx=0x7fac04009c20 comp rx=0 tx=0).stop 2024-05-04T12:09:52.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:52.270+0000 7fac18e3e700 1 -- 172.21.15.146:0/1460178992 shutdown_connections 2024-05-04T12:09:52.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:52.270+0000 7fac18e3e700 1 --2- 172.21.15.146:0/1460178992 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fac141008a0 0x7fac14100cc0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:09:52.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:52.270+0000 7fac18e3e700 1 -- 172.21.15.146:0/1460178992 >> 172.21.15.146:0/1460178992 conn(0x7fac140fbe20 msgr2=0x7fac140fe280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:09:52.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:52.270+0000 7fac18e3e700 1 -- 172.21.15.146:0/1460178992 shutdown_connections 2024-05-04T12:09:52.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:52.270+0000 7fac18e3e700 1 -- 172.21.15.146:0/1460178992 wait complete. 2024-05-04T12:09:52.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:52.271+0000 7fac18e3e700 1 Processor -- start 2024-05-04T12:09:52.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:52.271+0000 7fac18e3e700 1 -- start start 2024-05-04T12:09:52.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:52.271+0000 7fac18e3e700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fac141008a0 0x7fac14074b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:09:52.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:52.271+0000 7fac18e3e700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac14075070 con 0x7fac141008a0 2024-05-04T12:09:52.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:52.271+0000 7fac1259c700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fac141008a0 0x7fac14074b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:09:52.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:52.271+0000 7fac1259c700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fac141008a0 0x7fac14074b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.146:39956/0 (socket says 172.21.15.146:39956) 2024-05-04T12:09:52.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:52.271+0000 7fac1259c700 1 -- 172.21.15.146:0/3387496469 learned_addr learned my addr 172.21.15.146:0/3387496469 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-05-04T12:09:52.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:52.271+0000 7fac1259c700 1 -- 172.21.15.146:0/3387496469 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac040095f0 con 0x7fac141008a0 2024-05-04T12:09:52.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:52.272+0000 7fac1259c700 1 --2- 172.21.15.146:0/3387496469 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fac141008a0 0x7fac14074b30 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fac04005b40 tx=0x7fac04004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:09:52.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:52.272+0000 7fac037fe700 1 -- 172.21.15.146:0/3387496469 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fac0401d070 con 0x7fac141008a0 2024-05-04T12:09:52.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:52.272+0000 7fac18e3e700 1 -- 172.21.15.146:0/3387496469 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac14073180 con 0x7fac141008a0 2024-05-04T12:09:52.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:52.272+0000 7fac037fe700 1 -- 172.21.15.146:0/3387496469 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fac0400be10 con 0x7fac141008a0 2024-05-04T12:09:52.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:52.272+0000 7fac037fe700 1 -- 172.21.15.146:0/3387496469 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fac0400f460 con 0x7fac141008a0 2024-05-04T12:09:52.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:52.272+0000 7fac18e3e700 1 -- 172.21.15.146:0/3387496469 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac14073620 con 0x7fac141008a0 2024-05-04T12:09:52.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:52.273+0000 7fac18e3e700 1 -- 172.21.15.146:0/3387496469 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac140623c0 con 0x7fac141008a0 2024-05-04T12:09:52.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:52.273+0000 7fac037fe700 1 -- 172.21.15.146:0/3387496469 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 13) v1 ==== 45339+0+0 (secure 0 0 0) 0x7fac0400b960 con 0x7fac141008a0 2024-05-04T12:09:52.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:52.273+0000 7fac037fe700 1 --2- 172.21.15.146:0/3387496469 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7fabfc0383c0 0x7fabfc03a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:09:52.273 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:52.273+0000 7fac037fe700 1 -- 172.21.15.146:0/3387496469 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fac0404c480 con 0x7fac141008a0 2024-05-04T12:09:52.273 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:52.273+0000 7fac11d9b700 1 --2- 172.21.15.146:0/3387496469 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7fabfc0383c0 0x7fabfc03a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:09:52.273 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:52.273+0000 7fac11d9b700 1 --2- 172.21.15.146:0/3387496469 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7fabfc0383c0 0x7fabfc03a880 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fac08006fd0 tx=0x7fac08006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:09:52.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:52.282+0000 7fac037fe700 1 -- 172.21.15.146:0/3387496469 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fac0402a360 con 0x7fac141008a0 2024-05-04T12:09:52.543 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:52.544+0000 7fac18e3e700 1 -- 172.21.15.146:0/3387496469 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fac1418eea0 con 0x7fac141008a0 2024-05-04T12:09:52.543 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:52.545+0000 7fac037fe700 1 -- 172.21.15.146:0/3387496469 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7fac04027030 con 0x7fac141008a0 2024-05-04T12:09:52.544 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-05-04T12:09:52.544 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"d9b985c4-0a0e-11ef-bc97-c7b262605968","modified":"2024-05-04T12:07:48.391828Z","created":"2024-05-04T12:07:48.391828Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi079","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:3300","nonce":0},{"type":"v1","addr":"172.21.15.79:6789","nonce":0}]},"addr":"172.21.15.79:6789/0","public_addr":"172.21.15.79:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-05-04T12:09:52.545 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:52.547+0000 7fac18e3e700 1 -- 172.21.15.146:0/3387496469 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7fabfc0383c0 msgr2=0x7fabfc03a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:09:52.546 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:52.547+0000 7fac18e3e700 1 --2- 172.21.15.146:0/3387496469 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7fabfc0383c0 0x7fabfc03a880 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fac08006fd0 tx=0x7fac08006e40 comp rx=0 tx=0).stop 2024-05-04T12:09:52.546 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:52.547+0000 7fac18e3e700 1 -- 172.21.15.146:0/3387496469 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fac141008a0 msgr2=0x7fac14074b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:09:52.546 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:52.547+0000 7fac18e3e700 1 --2- 172.21.15.146:0/3387496469 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fac141008a0 0x7fac14074b30 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fac04005b40 tx=0x7fac04004740 comp rx=0 tx=0).stop 2024-05-04T12:09:52.546 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:52.547+0000 7fac18e3e700 1 -- 172.21.15.146:0/3387496469 shutdown_connections 2024-05-04T12:09:52.546 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:52.547+0000 7fac18e3e700 1 --2- 172.21.15.146:0/3387496469 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7fabfc0383c0 0x7fabfc03a880 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:09:52.546 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:52.547+0000 7fac18e3e700 1 --2- 172.21.15.146:0/3387496469 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fac141008a0 0x7fac14074b30 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:09:52.546 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:52.547+0000 7fac18e3e700 1 -- 172.21.15.146:0/3387496469 >> 172.21.15.146:0/3387496469 conn(0x7fac140fbe20 msgr2=0x7fac140fcb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:09:52.546 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:52.547+0000 7fac18e3e700 1 -- 172.21.15.146:0/3387496469 shutdown_connections 2024-05-04T12:09:52.546 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:52.547+0000 7fac18e3e700 1 -- 172.21.15.146:0/3387496469 wait complete. 2024-05-04T12:09:52.546 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-05-04T12:09:53.944 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:09:53 smithi079 ceph-mon[27222]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-04T12:09:53.945 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:09:53 smithi079 ceph-mon[27222]: from='client.? 172.21.15.146:0/3387496469' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-04T12:09:54.119 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-05-04T12:09:54.119 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d9b985c4-0a0e-11ef-bc97-c7b262605968 -- ceph mon dump -f json 2024-05-04T12:09:54.321 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /etc/ceph/ceph.conf 2024-05-04T12:09:54.365 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /etc/ceph/ceph.conf 2024-05-04T12:09:55.715 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:55.716+0000 7f8b41767700 1 -- 172.21.15.146:0/751535063 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8b3c0ff780 msgr2=0x7f8b3c0ffba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:09:55.715 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:55.716+0000 7f8b41767700 1 --2- 172.21.15.146:0/751535063 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8b3c0ff780 0x7f8b3c0ffba0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f8b2c009910 tx=0x7f8b2c009c20 comp rx=0 tx=0).stop 2024-05-04T12:09:55.715 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:55.716+0000 7f8b41767700 1 -- 172.21.15.146:0/751535063 shutdown_connections 2024-05-04T12:09:55.715 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:55.716+0000 7f8b41767700 1 --2- 172.21.15.146:0/751535063 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8b3c0ff780 0x7f8b3c0ffba0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:09:55.716 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:55.716+0000 7f8b41767700 1 -- 172.21.15.146:0/751535063 >> 172.21.15.146:0/751535063 conn(0x7f8b3c0fad20 msgr2=0x7f8b3c0fd160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:09:55.716 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:55.716+0000 7f8b41767700 1 -- 172.21.15.146:0/751535063 shutdown_connections 2024-05-04T12:09:55.716 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:55.716+0000 7f8b41767700 1 -- 172.21.15.146:0/751535063 wait complete. 2024-05-04T12:09:55.716 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:55.716+0000 7f8b41767700 1 Processor -- start 2024-05-04T12:09:55.716 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:55.717+0000 7f8b41767700 1 -- start start 2024-05-04T12:09:55.716 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:55.717+0000 7f8b41767700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8b3c0ff780 0x7f8b3c1959b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:09:55.716 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:55.717+0000 7f8b41767700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b3c195ef0 con 0x7f8b3c0ff780 2024-05-04T12:09:55.716 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:55.717+0000 7f8b3affd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8b3c0ff780 0x7f8b3c1959b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:09:55.716 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:55.717+0000 7f8b3affd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8b3c0ff780 0x7f8b3c1959b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.146:44888/0 (socket says 172.21.15.146:44888) 2024-05-04T12:09:55.716 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:55.717+0000 7f8b3affd700 1 -- 172.21.15.146:0/1901269640 learned_addr learned my addr 172.21.15.146:0/1901269640 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-05-04T12:09:55.716 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:55.717+0000 7f8b3affd700 1 -- 172.21.15.146:0/1901269640 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b2c0095f0 con 0x7f8b3c0ff780 2024-05-04T12:09:55.717 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:55.717+0000 7f8b3affd700 1 --2- 172.21.15.146:0/1901269640 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8b3c0ff780 0x7f8b3c1959b0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f8b2c005b40 tx=0x7f8b2c004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:09:55.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:55.718+0000 7f8b23fff700 1 -- 172.21.15.146:0/1901269640 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8b2c01d070 con 0x7f8b3c0ff780 2024-05-04T12:09:55.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:55.718+0000 7f8b23fff700 1 -- 172.21.15.146:0/1901269640 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f8b2c00be10 con 0x7f8b3c0ff780 2024-05-04T12:09:55.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:55.718+0000 7f8b41767700 1 -- 172.21.15.146:0/1901269640 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b3c1960f0 con 0x7f8b3c0ff780 2024-05-04T12:09:55.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:55.718+0000 7f8b23fff700 1 -- 172.21.15.146:0/1901269640 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8b2c00f460 con 0x7f8b3c0ff780 2024-05-04T12:09:55.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:55.718+0000 7f8b41767700 1 -- 172.21.15.146:0/1901269640 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b3c196590 con 0x7f8b3c0ff780 2024-05-04T12:09:55.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:55.719+0000 7f8b23fff700 1 -- 172.21.15.146:0/1901269640 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 13) v1 ==== 45339+0+0 (secure 0 0 0) 0x7f8b2c00f6d0 con 0x7f8b3c0ff780 2024-05-04T12:09:55.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:55.719+0000 7f8b41767700 1 -- 172.21.15.146:0/1901269640 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b3c1055c0 con 0x7f8b3c0ff780 2024-05-04T12:09:55.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:55.719+0000 7f8b23fff700 1 --2- 172.21.15.146:0/1901269640 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7f8b240383c0 0x7f8b2403a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:09:55.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:55.719+0000 7f8b23fff700 1 -- 172.21.15.146:0/1901269640 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f8b2c04c500 con 0x7f8b3c0ff780 2024-05-04T12:09:55.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:55.719+0000 7f8b3a7fc700 1 --2- 172.21.15.146:0/1901269640 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7f8b240383c0 0x7f8b2403a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:09:55.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:55.720+0000 7f8b3a7fc700 1 --2- 172.21.15.146:0/1901269640 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7f8b240383c0 0x7f8b2403a880 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f8b30006fd0 tx=0x7f8b30006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:09:55.722 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:55.724+0000 7f8b23fff700 1 -- 172.21.15.146:0/1901269640 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f8b2c020550 con 0x7f8b3c0ff780 2024-05-04T12:09:55.945 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:09:55 smithi079 ceph-mon[27222]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-04T12:09:55.989 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:55.990+0000 7f8b41767700 1 -- 172.21.15.146:0/1901269640 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8b3c199370 con 0x7f8b3c0ff780 2024-05-04T12:09:55.989 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:55.991+0000 7f8b23fff700 1 -- 172.21.15.146:0/1901269640 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f8b2c027030 con 0x7f8b3c0ff780 2024-05-04T12:09:55.989 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-05-04T12:09:55.989 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"d9b985c4-0a0e-11ef-bc97-c7b262605968","modified":"2024-05-04T12:07:48.391828Z","created":"2024-05-04T12:07:48.391828Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi079","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:3300","nonce":0},{"type":"v1","addr":"172.21.15.79:6789","nonce":0}]},"addr":"172.21.15.79:6789/0","public_addr":"172.21.15.79:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-05-04T12:09:55.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:55.993+0000 7f8b41767700 1 -- 172.21.15.146:0/1901269640 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7f8b240383c0 msgr2=0x7f8b2403a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:09:55.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:55.993+0000 7f8b41767700 1 --2- 172.21.15.146:0/1901269640 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7f8b240383c0 0x7f8b2403a880 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f8b30006fd0 tx=0x7f8b30006e40 comp rx=0 tx=0).stop 2024-05-04T12:09:55.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:55.993+0000 7f8b41767700 1 -- 172.21.15.146:0/1901269640 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8b3c0ff780 msgr2=0x7f8b3c1959b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:09:55.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:55.993+0000 7f8b41767700 1 --2- 172.21.15.146:0/1901269640 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8b3c0ff780 0x7f8b3c1959b0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f8b2c005b40 tx=0x7f8b2c004740 comp rx=0 tx=0).stop 2024-05-04T12:09:55.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:55.993+0000 7f8b41767700 1 -- 172.21.15.146:0/1901269640 shutdown_connections 2024-05-04T12:09:55.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:55.993+0000 7f8b41767700 1 --2- 172.21.15.146:0/1901269640 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7f8b240383c0 0x7f8b2403a880 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:09:55.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:55.993+0000 7f8b41767700 1 --2- 172.21.15.146:0/1901269640 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8b3c0ff780 0x7f8b3c1959b0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:09:55.993 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:55.993+0000 7f8b41767700 1 -- 172.21.15.146:0/1901269640 >> 172.21.15.146:0/1901269640 conn(0x7f8b3c0fad20 msgr2=0x7f8b3c0fba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:09:55.993 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:55.993+0000 7f8b41767700 1 -- 172.21.15.146:0/1901269640 shutdown_connections 2024-05-04T12:09:55.993 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:55.993+0000 7f8b41767700 1 -- 172.21.15.146:0/1901269640 wait complete. 2024-05-04T12:09:55.993 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-05-04T12:09:57.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:09:56 smithi079 ceph-mon[27222]: from='client.? 172.21.15.146:0/1901269640' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-04T12:09:57.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:09:56 smithi079 ceph-mon[27222]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-04T12:09:57.540 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-05-04T12:09:57.541 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d9b985c4-0a0e-11ef-bc97-c7b262605968 -- ceph mon dump -f json 2024-05-04T12:09:57.741 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /etc/ceph/ceph.conf 2024-05-04T12:09:57.786 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /etc/ceph/ceph.conf 2024-05-04T12:09:59.100 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:59.101+0000 7fb618eef700 1 -- 172.21.15.146:0/2198651066 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb614101310 msgr2=0x7fb614103700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:09:59.101 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:59.101+0000 7fb618eef700 1 --2- 172.21.15.146:0/2198651066 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb614101310 0x7fb614103700 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fb604009910 tx=0x7fb604009c20 comp rx=0 tx=0).stop 2024-05-04T12:09:59.101 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:59.101+0000 7fb618eef700 1 -- 172.21.15.146:0/2198651066 shutdown_connections 2024-05-04T12:09:59.101 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:59.101+0000 7fb618eef700 1 --2- 172.21.15.146:0/2198651066 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb614101310 0x7fb614103700 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:09:59.101 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:59.101+0000 7fb618eef700 1 -- 172.21.15.146:0/2198651066 >> 172.21.15.146:0/2198651066 conn(0x7fb6140fad20 msgr2=0x7fb6140fd160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:09:59.101 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:59.101+0000 7fb618eef700 1 -- 172.21.15.146:0/2198651066 shutdown_connections 2024-05-04T12:09:59.101 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:59.101+0000 7fb618eef700 1 -- 172.21.15.146:0/2198651066 wait complete. 2024-05-04T12:09:59.101 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:59.102+0000 7fb618eef700 1 Processor -- start 2024-05-04T12:09:59.101 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:59.102+0000 7fb618eef700 1 -- start start 2024-05-04T12:09:59.101 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:59.102+0000 7fb618eef700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb614101310 0x7fb614100c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:09:59.101 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:59.102+0000 7fb618eef700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6141011b0 con 0x7fb614101310 2024-05-04T12:09:59.103 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:59.103+0000 7fb61259c700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb614101310 0x7fb614100c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:09:59.103 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:59.103+0000 7fb61259c700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb614101310 0x7fb614100c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.146:44914/0 (socket says 172.21.15.146:44914) 2024-05-04T12:09:59.103 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:59.103+0000 7fb61259c700 1 -- 172.21.15.146:0/1686426292 learned_addr learned my addr 172.21.15.146:0/1686426292 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-05-04T12:09:59.103 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:59.103+0000 7fb61259c700 1 -- 172.21.15.146:0/1686426292 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6040095f0 con 0x7fb614101310 2024-05-04T12:09:59.103 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:59.103+0000 7fb61259c700 1 --2- 172.21.15.146:0/1686426292 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb614101310 0x7fb614100c70 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fb6040092e0 tx=0x7fb604009e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:09:59.103 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:59.103+0000 7fb60bfff700 1 -- 172.21.15.146:0/1686426292 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb60401d070 con 0x7fb614101310 2024-05-04T12:09:59.103 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:59.103+0000 7fb618eef700 1 -- 172.21.15.146:0/1686426292 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6140ff2c0 con 0x7fb614101310 2024-05-04T12:09:59.103 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:59.103+0000 7fb60bfff700 1 -- 172.21.15.146:0/1686426292 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fb60400be10 con 0x7fb614101310 2024-05-04T12:09:59.103 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:59.103+0000 7fb60bfff700 1 -- 172.21.15.146:0/1686426292 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb60400f460 con 0x7fb614101310 2024-05-04T12:09:59.103 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:59.103+0000 7fb618eef700 1 -- 172.21.15.146:0/1686426292 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6140ff760 con 0x7fb614101310 2024-05-04T12:09:59.103 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:59.104+0000 7fb60bfff700 1 -- 172.21.15.146:0/1686426292 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 13) v1 ==== 45339+0+0 (secure 0 0 0) 0x7fb60400f6d0 con 0x7fb614101310 2024-05-04T12:09:59.103 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:59.104+0000 7fb618eef700 1 -- 172.21.15.146:0/1686426292 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6140623c0 con 0x7fb614101310 2024-05-04T12:09:59.103 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:59.105+0000 7fb60bfff700 1 --2- 172.21.15.146:0/1686426292 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7fb6000383c0 0x7fb60003a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:09:59.104 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:59.105+0000 7fb60bfff700 1 -- 172.21.15.146:0/1686426292 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fb60404c7c0 con 0x7fb614101310 2024-05-04T12:09:59.104 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:59.105+0000 7fb609d9b700 1 --2- 172.21.15.146:0/1686426292 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7fb6000383c0 0x7fb60003a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:09:59.104 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:59.105+0000 7fb609d9b700 1 --2- 172.21.15.146:0/1686426292 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7fb6000383c0 0x7fb60003a880 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fb5fc006fd0 tx=0x7fb5fc006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:09:59.109 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:59.110+0000 7fb60bfff700 1 -- 172.21.15.146:0/1686426292 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fb604005c00 con 0x7fb614101310 2024-05-04T12:09:59.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:59.379+0000 7fb618eef700 1 -- 172.21.15.146:0/1686426292 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb61418ee80 con 0x7fb614101310 2024-05-04T12:09:59.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:59.379+0000 7fb60bfff700 1 -- 172.21.15.146:0/1686426292 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7fb604027020 con 0x7fb614101310 2024-05-04T12:09:59.378 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-05-04T12:09:59.378 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"d9b985c4-0a0e-11ef-bc97-c7b262605968","modified":"2024-05-04T12:07:48.391828Z","created":"2024-05-04T12:07:48.391828Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi079","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:3300","nonce":0},{"type":"v1","addr":"172.21.15.79:6789","nonce":0}]},"addr":"172.21.15.79:6789/0","public_addr":"172.21.15.79:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-05-04T12:09:59.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:59.381+0000 7fb618eef700 1 -- 172.21.15.146:0/1686426292 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7fb6000383c0 msgr2=0x7fb60003a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:09:59.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:59.381+0000 7fb618eef700 1 --2- 172.21.15.146:0/1686426292 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7fb6000383c0 0x7fb60003a880 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fb5fc006fd0 tx=0x7fb5fc006e40 comp rx=0 tx=0).stop 2024-05-04T12:09:59.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:59.382+0000 7fb618eef700 1 -- 172.21.15.146:0/1686426292 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb614101310 msgr2=0x7fb614100c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:09:59.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:59.382+0000 7fb618eef700 1 --2- 172.21.15.146:0/1686426292 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb614101310 0x7fb614100c70 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fb6040092e0 tx=0x7fb604009e60 comp rx=0 tx=0).stop 2024-05-04T12:09:59.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:59.382+0000 7fb618eef700 1 -- 172.21.15.146:0/1686426292 shutdown_connections 2024-05-04T12:09:59.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:59.382+0000 7fb618eef700 1 --2- 172.21.15.146:0/1686426292 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7fb6000383c0 0x7fb60003a880 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:09:59.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:59.382+0000 7fb618eef700 1 --2- 172.21.15.146:0/1686426292 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb614101310 0x7fb614100c70 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:09:59.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:59.382+0000 7fb618eef700 1 -- 172.21.15.146:0/1686426292 >> 172.21.15.146:0/1686426292 conn(0x7fb6140fad20 msgr2=0x7fb6140fb9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:09:59.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:59.382+0000 7fb618eef700 1 -- 172.21.15.146:0/1686426292 shutdown_connections 2024-05-04T12:09:59.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:09:59.382+0000 7fb618eef700 1 -- 172.21.15.146:0/1686426292 wait complete. 2024-05-04T12:09:59.381 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-05-04T12:09:59.695 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:09:59 smithi079 ceph-mon[27222]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-04T12:10:00.524 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:00 smithi079 ceph-mon[27222]: from='client.? 172.21.15.146:0/1686426292' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-04T12:10:00.943 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-05-04T12:10:00.944 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d9b985c4-0a0e-11ef-bc97-c7b262605968 -- ceph mon dump -f json 2024-05-04T12:10:01.147 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /etc/ceph/ceph.conf 2024-05-04T12:10:01.194 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /etc/ceph/ceph.conf 2024-05-04T12:10:01.653 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:01 smithi079 ceph-mon[27222]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-04T12:10:02.470 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:02.470+0000 7f1b0a4e4700 1 -- 172.21.15.146:0/3179907762 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1b041008a0 msgr2=0x7f1b04100cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:10:02.470 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:02.470+0000 7f1b0a4e4700 1 --2- 172.21.15.146:0/3179907762 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1b041008a0 0x7f1b04100cc0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f1af4009910 tx=0x7f1af4009c20 comp rx=0 tx=0).stop 2024-05-04T12:10:02.470 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:02.470+0000 7f1b0a4e4700 1 -- 172.21.15.146:0/3179907762 shutdown_connections 2024-05-04T12:10:02.470 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:02.470+0000 7f1b0a4e4700 1 --2- 172.21.15.146:0/3179907762 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1b041008a0 0x7f1b04100cc0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:10:02.470 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:02.470+0000 7f1b0a4e4700 1 -- 172.21.15.146:0/3179907762 >> 172.21.15.146:0/3179907762 conn(0x7f1b040fbe20 msgr2=0x7f1b040fe280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:10:02.470 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:02.470+0000 7f1b0a4e4700 1 -- 172.21.15.146:0/3179907762 shutdown_connections 2024-05-04T12:10:02.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:02.470+0000 7f1b0a4e4700 1 -- 172.21.15.146:0/3179907762 wait complete. 2024-05-04T12:10:02.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:02.470+0000 7f1b0a4e4700 1 Processor -- start 2024-05-04T12:10:02.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:02.471+0000 7f1b0a4e4700 1 -- start start 2024-05-04T12:10:02.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:02.471+0000 7f1b0a4e4700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1b041008a0 0x7f1b04074b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:10:02.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:02.471+0000 7f1b0a4e4700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b04075070 con 0x7f1b041008a0 2024-05-04T12:10:02.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:02.471+0000 7f1b03fff700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1b041008a0 0x7f1b04074b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:10:02.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:02.471+0000 7f1b03fff700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1b041008a0 0x7f1b04074b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.146:44934/0 (socket says 172.21.15.146:44934) 2024-05-04T12:10:02.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:02.471+0000 7f1b03fff700 1 -- 172.21.15.146:0/1026545571 learned_addr learned my addr 172.21.15.146:0/1026545571 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-05-04T12:10:02.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:02.471+0000 7f1b03fff700 1 -- 172.21.15.146:0/1026545571 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1af40095f0 con 0x7f1b041008a0 2024-05-04T12:10:02.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:02.471+0000 7f1b03fff700 1 --2- 172.21.15.146:0/1026545571 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1b041008a0 0x7f1b04074b30 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f1af4005950 tx=0x7f1af4009e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:10:02.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:02.472+0000 7f1b017fa700 1 -- 172.21.15.146:0/1026545571 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1af401d070 con 0x7f1b041008a0 2024-05-04T12:10:02.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:02.472+0000 7f1b017fa700 1 -- 172.21.15.146:0/1026545571 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f1af400be10 con 0x7f1b041008a0 2024-05-04T12:10:02.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:02.472+0000 7f1b0a4e4700 1 -- 172.21.15.146:0/1026545571 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b04073180 con 0x7f1b041008a0 2024-05-04T12:10:02.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:02.472+0000 7f1b017fa700 1 -- 172.21.15.146:0/1026545571 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1af400f460 con 0x7f1b041008a0 2024-05-04T12:10:02.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:02.472+0000 7f1b0a4e4700 1 -- 172.21.15.146:0/1026545571 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b04073620 con 0x7f1b041008a0 2024-05-04T12:10:02.473 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:02.472+0000 7f1b017fa700 1 -- 172.21.15.146:0/1026545571 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 13) v1 ==== 45339+0+0 (secure 0 0 0) 0x7f1af400f690 con 0x7f1b041008a0 2024-05-04T12:10:02.473 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:02.473+0000 7f1b017fa700 1 --2- 172.21.15.146:0/1026545571 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7f1aec0383c0 0x7f1aec03a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:10:02.473 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:02.473+0000 7f1b017fa700 1 -- 172.21.15.146:0/1026545571 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f1af404d590 con 0x7f1b041008a0 2024-05-04T12:10:02.473 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:02.473+0000 7f1b0a4e4700 1 -- 172.21.15.146:0/1026545571 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b040623c0 con 0x7f1b041008a0 2024-05-04T12:10:02.473 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:02.473+0000 7f1b037fe700 1 --2- 172.21.15.146:0/1026545571 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7f1aec0383c0 0x7f1aec03a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:10:02.473 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:02.473+0000 7f1b037fe700 1 --2- 172.21.15.146:0/1026545571 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7f1aec0383c0 0x7f1aec03a880 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f1af8006fd0 tx=0x7f1af8006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:10:02.476 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:02.478+0000 7f1b017fa700 1 -- 172.21.15.146:0/1026545571 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f1af4027070 con 0x7f1b041008a0 2024-05-04T12:10:02.737 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:02.738+0000 7f1b0a4e4700 1 -- 172.21.15.146:0/1026545571 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1b0419b9f0 con 0x7f1b041008a0 2024-05-04T12:10:02.737 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:02.739+0000 7f1b017fa700 1 -- 172.21.15.146:0/1026545571 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f1af402c440 con 0x7f1b041008a0 2024-05-04T12:10:02.737 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-05-04T12:10:02.738 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"d9b985c4-0a0e-11ef-bc97-c7b262605968","modified":"2024-05-04T12:07:48.391828Z","created":"2024-05-04T12:07:48.391828Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi079","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:3300","nonce":0},{"type":"v1","addr":"172.21.15.79:6789","nonce":0}]},"addr":"172.21.15.79:6789/0","public_addr":"172.21.15.79:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-05-04T12:10:02.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:02.741+0000 7f1b0a4e4700 1 -- 172.21.15.146:0/1026545571 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7f1aec0383c0 msgr2=0x7f1aec03a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:10:02.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:02.741+0000 7f1b0a4e4700 1 --2- 172.21.15.146:0/1026545571 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7f1aec0383c0 0x7f1aec03a880 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f1af8006fd0 tx=0x7f1af8006e40 comp rx=0 tx=0).stop 2024-05-04T12:10:02.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:02.741+0000 7f1b0a4e4700 1 -- 172.21.15.146:0/1026545571 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1b041008a0 msgr2=0x7f1b04074b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:10:02.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:02.741+0000 7f1b0a4e4700 1 --2- 172.21.15.146:0/1026545571 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1b041008a0 0x7f1b04074b30 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f1af4005950 tx=0x7f1af4009e60 comp rx=0 tx=0).stop 2024-05-04T12:10:02.740 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:02.741+0000 7f1b0a4e4700 1 -- 172.21.15.146:0/1026545571 shutdown_connections 2024-05-04T12:10:02.740 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:02.741+0000 7f1b0a4e4700 1 --2- 172.21.15.146:0/1026545571 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7f1aec0383c0 0x7f1aec03a880 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:10:02.740 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:02.741+0000 7f1b0a4e4700 1 --2- 172.21.15.146:0/1026545571 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1b041008a0 0x7f1b04074b30 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:10:02.740 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:02.741+0000 7f1b0a4e4700 1 -- 172.21.15.146:0/1026545571 >> 172.21.15.146:0/1026545571 conn(0x7f1b040fbe20 msgr2=0x7f1b040fcb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:10:02.740 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:02.741+0000 7f1b0a4e4700 1 -- 172.21.15.146:0/1026545571 shutdown_connections 2024-05-04T12:10:02.740 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:02.741+0000 7f1b0a4e4700 1 -- 172.21.15.146:0/1026545571 wait complete. 2024-05-04T12:10:02.740 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-05-04T12:10:03.647 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:03 smithi079 ceph-mon[27222]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-04T12:10:03.647 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:03 smithi079 ceph-mon[27222]: from='client.? 172.21.15.146:0/1026545571' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-04T12:10:04.286 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-05-04T12:10:04.286 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d9b985c4-0a0e-11ef-bc97-c7b262605968 -- ceph mon dump -f json 2024-05-04T12:10:04.487 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /etc/ceph/ceph.conf 2024-05-04T12:10:04.533 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /etc/ceph/ceph.conf 2024-05-04T12:10:05.386 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:05 smithi079 ceph-mon[27222]: from='mgr.14164 172.21.15.79:0/395052435' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:10:05.386 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:05 smithi079 ceph-mon[27222]: from='mgr.14164 172.21.15.79:0/395052435' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:10:05.386 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:05 smithi079 ceph-mon[27222]: from='mgr.14164 172.21.15.79:0/395052435' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:10:05.386 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:05 smithi079 ceph-mon[27222]: from='mgr.14164 172.21.15.79:0/395052435' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-05-04T12:10:05.387 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:05 smithi079 ceph-mon[27222]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-04T12:10:05.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:05.975+0000 7faf5db1b700 1 -- 172.21.15.146:0/891987884 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7faf58100d20 msgr2=0x7faf58101140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:10:05.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:05.975+0000 7faf5db1b700 1 --2- 172.21.15.146:0/891987884 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7faf58100d20 0x7faf58101140 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7faf4800b0a0 tx=0x7faf4800b3b0 comp rx=0 tx=0).stop 2024-05-04T12:10:05.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:05.975+0000 7faf5db1b700 1 -- 172.21.15.146:0/891987884 shutdown_connections 2024-05-04T12:10:05.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:05.975+0000 7faf5db1b700 1 --2- 172.21.15.146:0/891987884 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7faf58100d20 0x7faf58101140 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:10:05.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:05.975+0000 7faf5db1b700 1 -- 172.21.15.146:0/891987884 >> 172.21.15.146:0/891987884 conn(0x7faf580fa490 msgr2=0x7faf580fc8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:10:05.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:05.975+0000 7faf5db1b700 1 -- 172.21.15.146:0/891987884 shutdown_connections 2024-05-04T12:10:05.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:05.975+0000 7faf5db1b700 1 -- 172.21.15.146:0/891987884 wait complete. 2024-05-04T12:10:05.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:05.975+0000 7faf5db1b700 1 Processor -- start 2024-05-04T12:10:05.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:05.976+0000 7faf5db1b700 1 -- start start 2024-05-04T12:10:05.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:05.976+0000 7faf5db1b700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7faf58100d20 0x7faf58197ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:10:05.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:05.976+0000 7faf5db1b700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf581980e0 con 0x7faf58100d20 2024-05-04T12:10:05.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:05.976+0000 7faf577fe700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7faf58100d20 0x7faf58197ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:10:05.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:05.976+0000 7faf577fe700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7faf58100d20 0x7faf58197ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.146:47710/0 (socket says 172.21.15.146:47710) 2024-05-04T12:10:05.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:05.976+0000 7faf577fe700 1 -- 172.21.15.146:0/4091839866 learned_addr learned my addr 172.21.15.146:0/4091839866 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-05-04T12:10:05.977 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:05.976+0000 7faf577fe700 1 -- 172.21.15.146:0/4091839866 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf48009d00 con 0x7faf58100d20 2024-05-04T12:10:05.977 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:05.976+0000 7faf577fe700 1 --2- 172.21.15.146:0/4091839866 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7faf58100d20 0x7faf58197ba0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7faf48009fd0 tx=0x7faf4800bb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:10:05.977 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:05.977+0000 7faf54ff9700 1 -- 172.21.15.146:0/4091839866 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faf4800e040 con 0x7faf58100d20 2024-05-04T12:10:05.977 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:05.977+0000 7faf5db1b700 1 -- 172.21.15.146:0/4091839866 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf581982e0 con 0x7faf58100d20 2024-05-04T12:10:05.977 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:05.977+0000 7faf54ff9700 1 -- 172.21.15.146:0/4091839866 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7faf4800bdc0 con 0x7faf58100d20 2024-05-04T12:10:05.977 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:05.977+0000 7faf54ff9700 1 -- 172.21.15.146:0/4091839866 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faf4801da40 con 0x7faf58100d20 2024-05-04T12:10:05.977 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:05.977+0000 7faf5db1b700 1 -- 172.21.15.146:0/4091839866 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf58198700 con 0x7faf58100d20 2024-05-04T12:10:05.977 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:05.977+0000 7faf54ff9700 1 -- 172.21.15.146:0/4091839866 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 14) v1 ==== 45353+0+0 (secure 0 0 0) 0x7faf4801dc80 con 0x7faf58100d20 2024-05-04T12:10:05.977 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:05.977+0000 7faf5db1b700 1 -- 172.21.15.146:0/4091839866 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf581917c0 con 0x7faf58100d20 2024-05-04T12:10:05.977 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:05.978+0000 7faf54ff9700 1 --2- 172.21.15.146:0/4091839866 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7faf44038460 0x7faf4403a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:10:05.977 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:05.978+0000 7faf54ff9700 1 -- 172.21.15.146:0/4091839866 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7faf4804c8d0 con 0x7faf58100d20 2024-05-04T12:10:05.977 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:05.978+0000 7faf56ffd700 1 -- 172.21.15.146:0/4091839866 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7faf44038460 msgr2=0x7faf4403a920 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.79:6800/4263681324 2024-05-04T12:10:05.978 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:05.978+0000 7faf56ffd700 1 --2- 172.21.15.146:0/4091839866 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7faf44038460 0x7faf4403a920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-04T12:10:05.985 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:05.987+0000 7faf54ff9700 1 -- 172.21.15.146:0/4091839866 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7faf4801c5c0 con 0x7faf58100d20 2024-05-04T12:10:06.177 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:06.179+0000 7faf56ffd700 1 -- 172.21.15.146:0/4091839866 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7faf44038460 msgr2=0x7faf4403a920 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.79:6800/4263681324 2024-05-04T12:10:06.177 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:06.179+0000 7faf56ffd700 1 --2- 172.21.15.146:0/4091839866 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7faf44038460 0x7faf4403a920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-04T12:10:06.250 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:06.252+0000 7faf5db1b700 1 -- 172.21.15.146:0/4091839866 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7faf5802cc70 con 0x7faf58100d20 2024-05-04T12:10:06.251 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:06.253+0000 7faf54ff9700 1 -- 172.21.15.146:0/4091839866 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7faf4802c200 con 0x7faf58100d20 2024-05-04T12:10:06.251 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-05-04T12:10:06.251 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"d9b985c4-0a0e-11ef-bc97-c7b262605968","modified":"2024-05-04T12:07:48.391828Z","created":"2024-05-04T12:07:48.391828Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi079","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:3300","nonce":0},{"type":"v1","addr":"172.21.15.79:6789","nonce":0}]},"addr":"172.21.15.79:6789/0","public_addr":"172.21.15.79:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-05-04T12:10:06.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:06.255+0000 7faf5db1b700 1 -- 172.21.15.146:0/4091839866 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7faf44038460 msgr2=0x7faf4403a920 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-04T12:10:06.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:06.255+0000 7faf5db1b700 1 --2- 172.21.15.146:0/4091839866 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7faf44038460 0x7faf4403a920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:10:06.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:06.255+0000 7faf5db1b700 1 -- 172.21.15.146:0/4091839866 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7faf58100d20 msgr2=0x7faf58197ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:10:06.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:06.255+0000 7faf5db1b700 1 --2- 172.21.15.146:0/4091839866 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7faf58100d20 0x7faf58197ba0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7faf48009fd0 tx=0x7faf4800bb10 comp rx=0 tx=0).stop 2024-05-04T12:10:06.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:06.255+0000 7faf5db1b700 1 -- 172.21.15.146:0/4091839866 shutdown_connections 2024-05-04T12:10:06.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:06.255+0000 7faf5db1b700 1 --2- 172.21.15.146:0/4091839866 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7faf44038460 0x7faf4403a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:10:06.254 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:06.255+0000 7faf5db1b700 1 --2- 172.21.15.146:0/4091839866 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7faf58100d20 0x7faf58197ba0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:10:06.254 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:06.255+0000 7faf5db1b700 1 -- 172.21.15.146:0/4091839866 >> 172.21.15.146:0/4091839866 conn(0x7faf580fa490 msgr2=0x7faf580ff1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:10:06.254 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:06.255+0000 7faf5db1b700 1 -- 172.21.15.146:0/4091839866 shutdown_connections 2024-05-04T12:10:06.254 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:06.255+0000 7faf5db1b700 1 -- 172.21.15.146:0/4091839866 wait complete. 2024-05-04T12:10:06.254 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-05-04T12:10:06.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:06 smithi079 ceph-mon[27222]: from='mgr.14164 172.21.15.79:0/395052435' entity='mgr.smithi079.dgcpwl' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-05-04T12:10:06.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:06 smithi079 ceph-mon[27222]: mgrmap e14: smithi079.dgcpwl(active, since 66s) 2024-05-04T12:10:07.346 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:07 smithi079 ceph-mon[27222]: from='client.? 172.21.15.146:0/4091839866' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-04T12:10:07.793 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-05-04T12:10:07.793 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d9b985c4-0a0e-11ef-bc97-c7b262605968 -- ceph mon dump -f json 2024-05-04T12:10:07.993 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /etc/ceph/ceph.conf 2024-05-04T12:10:08.042 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /etc/ceph/ceph.conf 2024-05-04T12:10:09.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:09.569+0000 7f369b01a700 1 -- 172.21.15.146:0/3093319074 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f36941008a0 msgr2=0x7f3694100cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:10:09.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:09.569+0000 7f369b01a700 1 --2- 172.21.15.146:0/3093319074 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f36941008a0 0x7f3694100cc0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f368c00b0a0 tx=0x7f368c00b3b0 comp rx=0 tx=0).stop 2024-05-04T12:10:09.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:09.569+0000 7f369b01a700 1 -- 172.21.15.146:0/3093319074 shutdown_connections 2024-05-04T12:10:09.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:09.569+0000 7f369b01a700 1 --2- 172.21.15.146:0/3093319074 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f36941008a0 0x7f3694100cc0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:10:09.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:09.569+0000 7f369b01a700 1 -- 172.21.15.146:0/3093319074 >> 172.21.15.146:0/3093319074 conn(0x7f36940fbe20 msgr2=0x7f36940fe280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:10:09.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:09.569+0000 7f369b01a700 1 -- 172.21.15.146:0/3093319074 shutdown_connections 2024-05-04T12:10:09.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:09.569+0000 7f369b01a700 1 -- 172.21.15.146:0/3093319074 wait complete. 2024-05-04T12:10:09.571 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:09.570+0000 7f369b01a700 1 Processor -- start 2024-05-04T12:10:09.571 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:09.570+0000 7f369b01a700 1 -- start start 2024-05-04T12:10:09.571 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:09.570+0000 7f369b01a700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f36941008a0 0x7f36941959a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:10:09.571 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:09.570+0000 7f369b01a700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3694195ee0 con 0x7f36941008a0 2024-05-04T12:10:09.571 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:09.570+0000 7f3698db6700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f36941008a0 0x7f36941959a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:10:09.571 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:09.570+0000 7f3698db6700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f36941008a0 0x7f36941959a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.146:47736/0 (socket says 172.21.15.146:47736) 2024-05-04T12:10:09.571 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:09.570+0000 7f3698db6700 1 -- 172.21.15.146:0/3257666677 learned_addr learned my addr 172.21.15.146:0/3257666677 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-05-04T12:10:09.571 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:09.571+0000 7f3698db6700 1 -- 172.21.15.146:0/3257666677 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f368c009d00 con 0x7f36941008a0 2024-05-04T12:10:09.571 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:09.571+0000 7f3698db6700 1 --2- 172.21.15.146:0/3257666677 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f36941008a0 0x7f36941959a0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f368c00b830 tx=0x7f368c00bd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:10:09.572 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:09.571+0000 7f3691ffb700 1 -- 172.21.15.146:0/3257666677 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f368c00e040 con 0x7f36941008a0 2024-05-04T12:10:09.572 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:09.571+0000 7f369b01a700 1 -- 172.21.15.146:0/3257666677 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36941960e0 con 0x7f36941008a0 2024-05-04T12:10:09.572 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:09.571+0000 7f3691ffb700 1 -- 172.21.15.146:0/3257666677 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f368c012af0 con 0x7f36941008a0 2024-05-04T12:10:09.572 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:09.571+0000 7f3691ffb700 1 -- 172.21.15.146:0/3257666677 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f368c01d9f0 con 0x7f36941008a0 2024-05-04T12:10:09.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:09.571+0000 7f369b01a700 1 -- 172.21.15.146:0/3257666677 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3694196500 con 0x7f36941008a0 2024-05-04T12:10:09.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:09.572+0000 7f3691ffb700 1 -- 172.21.15.146:0/3257666677 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 14) v1 ==== 45353+0+0 (secure 0 0 0) 0x7f368c01dcb0 con 0x7f36941008a0 2024-05-04T12:10:09.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:09.572+0000 7f3691ffb700 1 --2- 172.21.15.146:0/3257666677 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7f3680038410 0x7f368003a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:10:09.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:09.572+0000 7f369b01a700 1 -- 172.21.15.146:0/3257666677 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f369418f4c0 con 0x7f36941008a0 2024-05-04T12:10:09.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:09.572+0000 7f3691ffb700 1 -- 172.21.15.146:0/3257666677 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f368c04cc70 con 0x7f36941008a0 2024-05-04T12:10:09.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:09.572+0000 7f3693fff700 1 -- 172.21.15.146:0/3257666677 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7f3680038410 msgr2=0x7f368003a8d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.79:6800/4263681324 2024-05-04T12:10:09.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:09.572+0000 7f3693fff700 1 --2- 172.21.15.146:0/3257666677 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7f3680038410 0x7f368003a8d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-04T12:10:09.579 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:09.580+0000 7f3691ffb700 1 -- 172.21.15.146:0/3257666677 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f368c01c3e0 con 0x7f36941008a0 2024-05-04T12:10:09.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:09.773+0000 7f3693fff700 1 -- 172.21.15.146:0/3257666677 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7f3680038410 msgr2=0x7f368003a8d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.79:6800/4263681324 2024-05-04T12:10:09.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:09.773+0000 7f3693fff700 1 --2- 172.21.15.146:0/3257666677 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7f3680038410 0x7f368003a8d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-04T12:10:09.840 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:09.841+0000 7f369b01a700 1 -- 172.21.15.146:0/3257666677 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f36940623c0 con 0x7f36941008a0 2024-05-04T12:10:09.841 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:09.842+0000 7f3691ffb700 1 -- 172.21.15.146:0/3257666677 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f368c01b340 con 0x7f36941008a0 2024-05-04T12:10:09.841 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-05-04T12:10:09.841 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"d9b985c4-0a0e-11ef-bc97-c7b262605968","modified":"2024-05-04T12:07:48.391828Z","created":"2024-05-04T12:07:48.391828Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi079","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:3300","nonce":0},{"type":"v1","addr":"172.21.15.79:6789","nonce":0}]},"addr":"172.21.15.79:6789/0","public_addr":"172.21.15.79:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-05-04T12:10:09.843 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:09.844+0000 7f369b01a700 1 -- 172.21.15.146:0/3257666677 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7f3680038410 msgr2=0x7f368003a8d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-04T12:10:09.843 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:09.844+0000 7f369b01a700 1 --2- 172.21.15.146:0/3257666677 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7f3680038410 0x7f368003a8d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:10:09.843 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:09.844+0000 7f369b01a700 1 -- 172.21.15.146:0/3257666677 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f36941008a0 msgr2=0x7f36941959a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:10:09.843 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:09.844+0000 7f369b01a700 1 --2- 172.21.15.146:0/3257666677 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f36941008a0 0x7f36941959a0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f368c00b830 tx=0x7f368c00bd40 comp rx=0 tx=0).stop 2024-05-04T12:10:09.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:09.844+0000 7f369b01a700 1 -- 172.21.15.146:0/3257666677 shutdown_connections 2024-05-04T12:10:09.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:09.844+0000 7f369b01a700 1 --2- 172.21.15.146:0/3257666677 >> [v2:172.21.15.79:6800/4263681324,v1:172.21.15.79:6801/4263681324] conn(0x7f3680038410 0x7f368003a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:10:09.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:09.844+0000 7f369b01a700 1 --2- 172.21.15.146:0/3257666677 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f36941008a0 0x7f36941959a0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:10:09.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:09.844+0000 7f369b01a700 1 -- 172.21.15.146:0/3257666677 >> 172.21.15.146:0/3257666677 conn(0x7f36940fbe20 msgr2=0x7f36940fcb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:10:09.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:09.844+0000 7f369b01a700 1 -- 172.21.15.146:0/3257666677 shutdown_connections 2024-05-04T12:10:09.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:09.844+0000 7f369b01a700 1 -- 172.21.15.146:0/3257666677 wait complete. 2024-05-04T12:10:09.844 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-05-04T12:10:10.165 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:09 smithi079 ceph-mon[27222]: from='client.? 172.21.15.146:0/3257666677' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-04T12:10:11.362 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-05-04T12:10:11.363 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d9b985c4-0a0e-11ef-bc97-c7b262605968 -- ceph mon dump -f json 2024-05-04T12:10:11.563 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /etc/ceph/ceph.conf 2024-05-04T12:10:11.610 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /etc/ceph/ceph.conf 2024-05-04T12:10:12.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:11 smithi079 ceph-mon[27222]: Active manager daemon smithi079.dgcpwl restarted 2024-05-04T12:10:12.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:11 smithi079 ceph-mon[27222]: Activating manager daemon smithi079.dgcpwl 2024-05-04T12:10:12.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:11 smithi079 ceph-mon[27222]: osdmap e5: 0 total, 0 up, 0 in 2024-05-04T12:10:12.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:11 smithi079 ceph-mon[27222]: mgrmap e15: smithi079.dgcpwl(active, starting, since 0.00295176s) 2024-05-04T12:10:12.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:11 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "mon metadata", "id": "smithi079"}]: dispatch 2024-05-04T12:10:12.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:11 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "mgr metadata", "who": "smithi079.dgcpwl", "id": "smithi079.dgcpwl"}]: dispatch 2024-05-04T12:10:12.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:11 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-04T12:10:12.446 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:11 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-04T12:10:12.446 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:11 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-04T12:10:12.446 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:11 smithi079 ceph-mon[27222]: Manager daemon smithi079.dgcpwl is now available 2024-05-04T12:10:12.446 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:11 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-04T12:10:12.446 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:11 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-04T12:10:12.446 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:11 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:10:12.446 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:11 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-04T12:10:12.446 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:11 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi079.dgcpwl/mirror_snapshot_schedule"}]: dispatch 2024-05-04T12:10:12.446 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:11 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi079.dgcpwl/trash_purge_schedule"}]: dispatch 2024-05-04T12:10:13.134 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:13.133+0000 7f4ef2896700 1 -- 172.21.15.146:0/4039124289 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4eec1008a0 msgr2=0x7f4eec100cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:10:13.134 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:13.133+0000 7f4ef2896700 1 --2- 172.21.15.146:0/4039124289 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4eec1008a0 0x7f4eec100cc0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f4edc009910 tx=0x7f4edc009c20 comp rx=0 tx=0).stop 2024-05-04T12:10:13.134 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:13.133+0000 7f4ef2896700 1 -- 172.21.15.146:0/4039124289 shutdown_connections 2024-05-04T12:10:13.134 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:13.133+0000 7f4ef2896700 1 --2- 172.21.15.146:0/4039124289 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4eec1008a0 0x7f4eec100cc0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:10:13.134 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:13.133+0000 7f4ef2896700 1 -- 172.21.15.146:0/4039124289 >> 172.21.15.146:0/4039124289 conn(0x7f4eec0fbe20 msgr2=0x7f4eec0fe280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:10:13.134 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:13.133+0000 7f4ef2896700 1 -- 172.21.15.146:0/4039124289 shutdown_connections 2024-05-04T12:10:13.134 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:13.133+0000 7f4ef2896700 1 -- 172.21.15.146:0/4039124289 wait complete. 2024-05-04T12:10:13.134 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:13.134+0000 7f4ef2896700 1 Processor -- start 2024-05-04T12:10:13.134 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:13.134+0000 7f4ef2896700 1 -- start start 2024-05-04T12:10:13.134 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:13.134+0000 7f4ef2896700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4eec1008a0 0x7f4eec1959f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:10:13.135 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:13.134+0000 7f4ef2896700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4eec195f30 con 0x7f4eec1008a0 2024-05-04T12:10:13.135 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:13.135+0000 7f4eebfff700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4eec1008a0 0x7f4eec1959f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:10:13.135 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:13.135+0000 7f4eebfff700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4eec1008a0 0x7f4eec1959f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.146:47762/0 (socket says 172.21.15.146:47762) 2024-05-04T12:10:13.135 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:13.135+0000 7f4eebfff700 1 -- 172.21.15.146:0/1898060410 learned_addr learned my addr 172.21.15.146:0/1898060410 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-05-04T12:10:13.136 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:13.135+0000 7f4eebfff700 1 -- 172.21.15.146:0/1898060410 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4edc0095f0 con 0x7f4eec1008a0 2024-05-04T12:10:13.136 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:13.135+0000 7f4eebfff700 1 --2- 172.21.15.146:0/1898060410 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4eec1008a0 0x7f4eec1959f0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f4edc005b40 tx=0x7f4edc004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:10:13.136 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:13.135+0000 7f4ee97fa700 1 -- 172.21.15.146:0/1898060410 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4edc01d070 con 0x7f4eec1008a0 2024-05-04T12:10:13.136 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:13.135+0000 7f4ee97fa700 1 -- 172.21.15.146:0/1898060410 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f4edc00be10 con 0x7f4eec1008a0 2024-05-04T12:10:13.136 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:13.135+0000 7f4ef2896700 1 -- 172.21.15.146:0/1898060410 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4eec196130 con 0x7f4eec1008a0 2024-05-04T12:10:13.137 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:13.135+0000 7f4ee97fa700 1 -- 172.21.15.146:0/1898060410 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4edc00f460 con 0x7f4eec1008a0 2024-05-04T12:10:13.137 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:13.135+0000 7f4ef2896700 1 -- 172.21.15.146:0/1898060410 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4eec1965d0 con 0x7f4eec1008a0 2024-05-04T12:10:13.137 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:13.136+0000 7f4ee97fa700 1 -- 172.21.15.146:0/1898060410 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 16) v1 ==== 45204+0+0 (secure 0 0 0) 0x7f4edc00f6d0 con 0x7f4eec1008a0 2024-05-04T12:10:13.137 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:13.136+0000 7f4ee97fa700 1 --2- 172.21.15.146:0/1898060410 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f4ed4038270 0x7f4ed403a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:10:13.137 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:13.136+0000 7f4ee97fa700 1 -- 172.21.15.146:0/1898060410 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7f4edc04d370 con 0x7f4eec1008a0 2024-05-04T12:10:13.137 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:13.137+0000 7f4ef2896700 1 -- 172.21.15.146:0/1898060410 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4eec18f4c0 con 0x7f4eec1008a0 2024-05-04T12:10:13.137 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:13.137+0000 7f4eeb7fe700 1 --2- 172.21.15.146:0/1898060410 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f4ed4038270 0x7f4ed403a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:10:13.137 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:13.137+0000 7f4eeb7fe700 1 --2- 172.21.15.146:0/1898060410 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f4ed4038270 0x7f4ed403a730 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f4ee0006fd0 tx=0x7f4ee0006e40 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:10:13.140 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:13.141+0000 7f4ee97fa700 1 -- 172.21.15.146:0/1898060410 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f4edc027030 con 0x7f4eec1008a0 2024-05-04T12:10:13.417 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:13.417+0000 7f4ef2896700 1 -- 172.21.15.146:0/1898060410 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4eec0623c0 con 0x7f4eec1008a0 2024-05-04T12:10:13.417 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:13.418+0000 7f4ee97fa700 1 -- 172.21.15.146:0/1898060410 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f4edc02c440 con 0x7f4eec1008a0 2024-05-04T12:10:13.417 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-05-04T12:10:13.417 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"d9b985c4-0a0e-11ef-bc97-c7b262605968","modified":"2024-05-04T12:07:48.391828Z","created":"2024-05-04T12:07:48.391828Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi079","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:3300","nonce":0},{"type":"v1","addr":"172.21.15.79:6789","nonce":0}]},"addr":"172.21.15.79:6789/0","public_addr":"172.21.15.79:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-05-04T12:10:13.419 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:13.420+0000 7f4ef2896700 1 -- 172.21.15.146:0/1898060410 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f4ed4038270 msgr2=0x7f4ed403a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:10:13.419 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:13.420+0000 7f4ef2896700 1 --2- 172.21.15.146:0/1898060410 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f4ed4038270 0x7f4ed403a730 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f4ee0006fd0 tx=0x7f4ee0006e40 comp rx=0 tx=0).stop 2024-05-04T12:10:13.419 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:13.420+0000 7f4ef2896700 1 -- 172.21.15.146:0/1898060410 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4eec1008a0 msgr2=0x7f4eec1959f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:10:13.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:13.420+0000 7f4ef2896700 1 --2- 172.21.15.146:0/1898060410 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4eec1008a0 0x7f4eec1959f0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f4edc005b40 tx=0x7f4edc004740 comp rx=0 tx=0).stop 2024-05-04T12:10:13.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:13.420+0000 7f4ef2896700 1 -- 172.21.15.146:0/1898060410 shutdown_connections 2024-05-04T12:10:13.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:13.420+0000 7f4ef2896700 1 --2- 172.21.15.146:0/1898060410 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f4ed4038270 0x7f4ed403a730 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:10:13.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:13.420+0000 7f4ef2896700 1 --2- 172.21.15.146:0/1898060410 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4eec1008a0 0x7f4eec1959f0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:10:13.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:13.420+0000 7f4ef2896700 1 -- 172.21.15.146:0/1898060410 >> 172.21.15.146:0/1898060410 conn(0x7f4eec0fbe20 msgr2=0x7f4eec0fcb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:10:13.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:13.420+0000 7f4ef2896700 1 -- 172.21.15.146:0/1898060410 shutdown_connections 2024-05-04T12:10:13.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:13.420+0000 7f4ef2896700 1 -- 172.21.15.146:0/1898060410 wait complete. 2024-05-04T12:10:13.420 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-05-04T12:10:13.694 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:13 smithi079 ceph-mon[27222]: [04/May/2024:12:10:12] ENGINE Bus STARTING 2024-05-04T12:10:13.695 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:13 smithi079 ceph-mon[27222]: [04/May/2024:12:10:12] ENGINE Serving on https://172.21.15.79:7150 2024-05-04T12:10:13.695 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:13 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:10:13.695 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:13 smithi079 ceph-mon[27222]: [04/May/2024:12:10:12] ENGINE Serving on http://172.21.15.79:8765 2024-05-04T12:10:13.695 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:13 smithi079 ceph-mon[27222]: [04/May/2024:12:10:12] ENGINE Bus STARTED 2024-05-04T12:10:13.695 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:13 smithi079 ceph-mon[27222]: mgrmap e16: smithi079.dgcpwl(active, since 1.00563s) 2024-05-04T12:10:13.695 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:13 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:10:14.660 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:14 smithi079 ceph-mon[27222]: from='client.? 172.21.15.146:0/1898060410' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-04T12:10:14.661 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:14 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:10:15.002 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-05-04T12:10:15.002 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d9b985c4-0a0e-11ef-bc97-c7b262605968 -- ceph mon dump -f json 2024-05-04T12:10:15.521 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /etc/ceph/ceph.conf 2024-05-04T12:10:15.574 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:15 smithi079 ceph-mon[27222]: mgrmap e17: smithi079.dgcpwl(active, since 2s) 2024-05-04T12:10:15.575 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /etc/ceph/ceph.conf 2024-05-04T12:10:17.195 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:16 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:10:17.195 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:16 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:10:17.195 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:16 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:10:17.196 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:16 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:10:17.196 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:16 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config rm", "who": "osd/host:smithi079", "name": "osd_memory_target"}]: dispatch 2024-05-04T12:10:17.196 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:16 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:10:17.196 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:16 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:10:17.196 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:16 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:10:17.196 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:16 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:10:17.196 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:16 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config rm", "who": "osd/host:smithi146", "name": "osd_memory_target"}]: dispatch 2024-05-04T12:10:17.196 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:16 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-04T12:10:17.196 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:16 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-04T12:10:17.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:17.603+0000 7f347331c700 1 -- 172.21.15.146:0/3336678588 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f346c0ff780 msgr2=0x7f346c0ffba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:10:17.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:17.603+0000 7f347331c700 1 --2- 172.21.15.146:0/3336678588 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f346c0ff780 0x7f346c0ffba0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f3468009910 tx=0x7f3468009c20 comp rx=0 tx=0).stop 2024-05-04T12:10:17.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:17.603+0000 7f347331c700 1 -- 172.21.15.146:0/3336678588 shutdown_connections 2024-05-04T12:10:17.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:17.603+0000 7f347331c700 1 --2- 172.21.15.146:0/3336678588 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f346c0ff780 0x7f346c0ffba0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:10:17.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:17.603+0000 7f347331c700 1 -- 172.21.15.146:0/3336678588 >> 172.21.15.146:0/3336678588 conn(0x7f346c0fad00 msgr2=0x7f346c0fd160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:10:17.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:17.603+0000 7f347331c700 1 -- 172.21.15.146:0/3336678588 shutdown_connections 2024-05-04T12:10:17.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:17.603+0000 7f347331c700 1 -- 172.21.15.146:0/3336678588 wait complete. 2024-05-04T12:10:17.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:17.603+0000 7f347331c700 1 Processor -- start 2024-05-04T12:10:17.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:17.603+0000 7f347331c700 1 -- start start 2024-05-04T12:10:17.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:17.603+0000 7f347331c700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f346c0ff780 0x7f346c197bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:10:17.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:17.603+0000 7f347331c700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f346c198100 con 0x7f346c0ff780 2024-05-04T12:10:17.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:17.604+0000 7f34710b8700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f346c0ff780 0x7f346c197bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:10:17.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:17.604+0000 7f34710b8700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f346c0ff780 0x7f346c197bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.146:45514/0 (socket says 172.21.15.146:45514) 2024-05-04T12:10:17.606 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:17.604+0000 7f34710b8700 1 -- 172.21.15.146:0/1030243821 learned_addr learned my addr 172.21.15.146:0/1030243821 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-05-04T12:10:17.606 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:17.604+0000 7f34710b8700 1 -- 172.21.15.146:0/1030243821 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34680095f0 con 0x7f346c0ff780 2024-05-04T12:10:17.606 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:17.604+0000 7f34710b8700 1 --2- 172.21.15.146:0/1030243821 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f346c0ff780 0x7f346c197bc0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f3468005b40 tx=0x7f3468004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:10:17.606 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:17.604+0000 7f34627fc700 1 -- 172.21.15.146:0/1030243821 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f346801d070 con 0x7f346c0ff780 2024-05-04T12:10:17.606 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:17.604+0000 7f34627fc700 1 -- 172.21.15.146:0/1030243821 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f346800be10 con 0x7f346c0ff780 2024-05-04T12:10:17.606 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:17.604+0000 7f347331c700 1 -- 172.21.15.146:0/1030243821 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f346c198300 con 0x7f346c0ff780 2024-05-04T12:10:17.606 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:17.604+0000 7f34627fc700 1 -- 172.21.15.146:0/1030243821 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f346800f460 con 0x7f346c0ff780 2024-05-04T12:10:17.606 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:17.604+0000 7f347331c700 1 -- 172.21.15.146:0/1030243821 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f346c1987a0 con 0x7f346c0ff780 2024-05-04T12:10:17.606 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:17.605+0000 7f34627fc700 1 -- 172.21.15.146:0/1030243821 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 17) v1 ==== 45396+0+0 (secure 0 0 0) 0x7f34680227e0 con 0x7f346c0ff780 2024-05-04T12:10:17.606 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:17.605+0000 7f347331c700 1 -- 172.21.15.146:0/1030243821 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f346c1917b0 con 0x7f346c0ff780 2024-05-04T12:10:17.606 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:17.605+0000 7f34627fc700 1 --2- 172.21.15.146:0/1030243821 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f34580384b0 0x7f345803a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:10:17.607 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:17.605+0000 7f34627fc700 1 -- 172.21.15.146:0/1030243821 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7f346804c670 con 0x7f346c0ff780 2024-05-04T12:10:17.607 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:17.605+0000 7f34708b7700 1 --2- 172.21.15.146:0/1030243821 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f34580384b0 0x7f345803a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:10:17.607 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:17.606+0000 7f34708b7700 1 --2- 172.21.15.146:0/1030243821 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f34580384b0 0x7f345803a970 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f345c006fd0 tx=0x7f345c006e40 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:10:17.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:17.614+0000 7f34627fc700 1 -- 172.21.15.146:0/1030243821 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f346802bba0 con 0x7f346c0ff780 2024-05-04T12:10:17.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:17.894+0000 7f347331c700 1 -- 172.21.15.146:0/1030243821 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f346c02cc70 con 0x7f346c0ff780 2024-05-04T12:10:17.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:17.895+0000 7f34627fc700 1 -- 172.21.15.146:0/1030243821 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f3468027030 con 0x7f346c0ff780 2024-05-04T12:10:17.894 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-05-04T12:10:17.894 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"d9b985c4-0a0e-11ef-bc97-c7b262605968","modified":"2024-05-04T12:07:48.391828Z","created":"2024-05-04T12:07:48.391828Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi079","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:3300","nonce":0},{"type":"v1","addr":"172.21.15.79:6789","nonce":0}]},"addr":"172.21.15.79:6789/0","public_addr":"172.21.15.79:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-05-04T12:10:17.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:17.897+0000 7f347331c700 1 -- 172.21.15.146:0/1030243821 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f34580384b0 msgr2=0x7f345803a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:10:17.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:17.897+0000 7f347331c700 1 --2- 172.21.15.146:0/1030243821 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f34580384b0 0x7f345803a970 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f345c006fd0 tx=0x7f345c006e40 comp rx=0 tx=0).stop 2024-05-04T12:10:17.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:17.897+0000 7f347331c700 1 -- 172.21.15.146:0/1030243821 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f346c0ff780 msgr2=0x7f346c197bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:10:17.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:17.897+0000 7f347331c700 1 --2- 172.21.15.146:0/1030243821 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f346c0ff780 0x7f346c197bc0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f3468005b40 tx=0x7f3468004740 comp rx=0 tx=0).stop 2024-05-04T12:10:17.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:17.897+0000 7f347331c700 1 -- 172.21.15.146:0/1030243821 shutdown_connections 2024-05-04T12:10:17.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:17.897+0000 7f347331c700 1 --2- 172.21.15.146:0/1030243821 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f34580384b0 0x7f345803a970 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:10:17.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:17.897+0000 7f347331c700 1 --2- 172.21.15.146:0/1030243821 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f346c0ff780 0x7f346c197bc0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:10:17.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:17.897+0000 7f347331c700 1 -- 172.21.15.146:0/1030243821 >> 172.21.15.146:0/1030243821 conn(0x7f346c0fad00 msgr2=0x7f346c069090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:10:17.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:17.897+0000 7f347331c700 1 -- 172.21.15.146:0/1030243821 shutdown_connections 2024-05-04T12:10:17.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:17.897+0000 7f347331c700 1 -- 172.21.15.146:0/1030243821 wait complete. 2024-05-04T12:10:17.897 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-05-04T12:10:18.195 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:17 smithi079 ceph-mon[27222]: Updating smithi079:/etc/ceph/ceph.conf 2024-05-04T12:10:18.195 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:17 smithi079 ceph-mon[27222]: Updating smithi146:/etc/ceph/ceph.conf 2024-05-04T12:10:18.195 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:17 smithi079 ceph-mon[27222]: Updating smithi079:/var/lib/ceph/d9b985c4-0a0e-11ef-bc97-c7b262605968/config/ceph.conf 2024-05-04T12:10:18.195 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:17 smithi079 ceph-mon[27222]: Updating smithi146:/var/lib/ceph/d9b985c4-0a0e-11ef-bc97-c7b262605968/config/ceph.conf 2024-05-04T12:10:18.195 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:17 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:10:18.195 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:17 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:10:18.196 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:17 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:10:18.196 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:17 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:10:18.196 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:17 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:10:18.196 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:17 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi146", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-05-04T12:10:18.196 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:17 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd='[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi146", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]': finished 2024-05-04T12:10:18.196 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:17 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-04T12:10:18.196 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:17 smithi079 ceph-mon[27222]: from='client.? 172.21.15.146:0/1030243821' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-04T12:10:19.195 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:18 smithi079 ceph-mon[27222]: Updating smithi146:/etc/ceph/ceph.client.admin.keyring 2024-05-04T12:10:19.195 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:18 smithi079 ceph-mon[27222]: Updating smithi079:/etc/ceph/ceph.client.admin.keyring 2024-05-04T12:10:19.195 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:18 smithi079 ceph-mon[27222]: Updating smithi146:/var/lib/ceph/d9b985c4-0a0e-11ef-bc97-c7b262605968/config/ceph.client.admin.keyring 2024-05-04T12:10:19.195 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:18 smithi079 ceph-mon[27222]: Updating smithi079:/var/lib/ceph/d9b985c4-0a0e-11ef-bc97-c7b262605968/config/ceph.client.admin.keyring 2024-05-04T12:10:19.195 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:18 smithi079 ceph-mon[27222]: Deploying daemon ceph-exporter.smithi146 on smithi146 2024-05-04T12:10:19.528 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-05-04T12:10:19.528 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d9b985c4-0a0e-11ef-bc97-c7b262605968 -- ceph mon dump -f json 2024-05-04T12:10:19.809 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/d9b985c4-0a0e-11ef-bc97-c7b262605968/config/ceph.conf 2024-05-04T12:10:21.749 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:21.616+0000 7fc5bdd3c700 1 -- 172.21.15.146:0/450304088 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc5b80ff7d0 msgr2=0x7fc5b80ffbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:10:21.749 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:21.616+0000 7fc5bdd3c700 1 --2- 172.21.15.146:0/450304088 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc5b80ff7d0 0x7fc5b80ffbb0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fc5a8009910 tx=0x7fc5a8009c20 comp rx=0 tx=0).stop 2024-05-04T12:10:21.749 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:21.616+0000 7fc5bdd3c700 1 -- 172.21.15.146:0/450304088 shutdown_connections 2024-05-04T12:10:21.749 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:21.616+0000 7fc5bdd3c700 1 --2- 172.21.15.146:0/450304088 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc5b80ff7d0 0x7fc5b80ffbb0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:10:21.749 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:21.616+0000 7fc5bdd3c700 1 -- 172.21.15.146:0/450304088 >> 172.21.15.146:0/450304088 conn(0x7fc5b80fb250 msgr2=0x7fc5b80fd670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:10:21.749 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:21.616+0000 7fc5bdd3c700 1 -- 172.21.15.146:0/450304088 shutdown_connections 2024-05-04T12:10:21.749 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:21.616+0000 7fc5bdd3c700 1 -- 172.21.15.146:0/450304088 wait complete. 2024-05-04T12:10:21.749 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:21.617+0000 7fc5bdd3c700 1 Processor -- start 2024-05-04T12:10:21.749 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:21.617+0000 7fc5bdd3c700 1 -- start start 2024-05-04T12:10:21.749 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:21.617+0000 7fc5bdd3c700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc5b80ff7d0 0x7fc5b819c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:10:21.749 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:21.617+0000 7fc5bdd3c700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5b819c910 con 0x7fc5b80ff7d0 2024-05-04T12:10:21.750 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:21.618+0000 7fc5b77fe700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc5b80ff7d0 0x7fc5b819c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:10:21.750 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:21.618+0000 7fc5b77fe700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc5b80ff7d0 0x7fc5b819c3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.146:45530/0 (socket says 172.21.15.146:45530) 2024-05-04T12:10:21.750 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:21.618+0000 7fc5b77fe700 1 -- 172.21.15.146:0/450420403 learned_addr learned my addr 172.21.15.146:0/450420403 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-05-04T12:10:21.750 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:21.618+0000 7fc5b77fe700 1 -- 172.21.15.146:0/450420403 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5a80095f0 con 0x7fc5b80ff7d0 2024-05-04T12:10:21.750 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:21.618+0000 7fc5b77fe700 1 --2- 172.21.15.146:0/450420403 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc5b80ff7d0 0x7fc5b819c3d0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fc5a8005b40 tx=0x7fc5a8004c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:10:21.750 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:21.618+0000 7fc5b4ff9700 1 -- 172.21.15.146:0/450420403 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc5a801d070 con 0x7fc5b80ff7d0 2024-05-04T12:10:21.750 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:21.618+0000 7fc5b4ff9700 1 -- 172.21.15.146:0/450420403 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fc5a80044d0 con 0x7fc5b80ff7d0 2024-05-04T12:10:21.750 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:21.618+0000 7fc5bdd3c700 1 -- 172.21.15.146:0/450420403 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5b819cb10 con 0x7fc5b80ff7d0 2024-05-04T12:10:21.750 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:21.618+0000 7fc5b4ff9700 1 -- 172.21.15.146:0/450420403 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc5a8003930 con 0x7fc5b80ff7d0 2024-05-04T12:10:21.750 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:21.618+0000 7fc5bdd3c700 1 -- 172.21.15.146:0/450420403 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5b819ced0 con 0x7fc5b80ff7d0 2024-05-04T12:10:21.750 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:21.619+0000 7fc5b4ff9700 1 -- 172.21.15.146:0/450420403 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 17) v1 ==== 45396+0+0 (secure 0 0 0) 0x7fc5a8003bf0 con 0x7fc5b80ff7d0 2024-05-04T12:10:21.750 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:21.619+0000 7fc5bdd3c700 1 -- 172.21.15.146:0/450420403 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5b80623c0 con 0x7fc5b80ff7d0 2024-05-04T12:10:21.750 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:21.619+0000 7fc5b4ff9700 1 --2- 172.21.15.146:0/450420403 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7fc5a00384b0 0x7fc5a003a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:10:21.750 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:21.619+0000 7fc5b4ff9700 1 -- 172.21.15.146:0/450420403 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7fc5a804c3a0 con 0x7fc5b80ff7d0 2024-05-04T12:10:21.751 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:21.620+0000 7fc5b6ffd700 1 --2- 172.21.15.146:0/450420403 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7fc5a00384b0 0x7fc5a003a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:10:21.751 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:21.620+0000 7fc5b6ffd700 1 --2- 172.21.15.146:0/450420403 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7fc5a00384b0 0x7fc5a003a970 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fc5ac006fd0 tx=0x7fc5ac006e40 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:10:21.751 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:21.624+0000 7fc5b4ff9700 1 -- 172.21.15.146:0/450420403 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fc5a802a540 con 0x7fc5b80ff7d0 2024-05-04T12:10:21.901 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:21.901+0000 7fc5bdd3c700 1 -- 172.21.15.146:0/450420403 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc5b8103810 con 0x7fc5b80ff7d0 2024-05-04T12:10:21.902 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:21.902+0000 7fc5b4ff9700 1 -- 172.21.15.146:0/450420403 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7fc5a802a360 con 0x7fc5b80ff7d0 2024-05-04T12:10:21.902 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-05-04T12:10:21.903 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"d9b985c4-0a0e-11ef-bc97-c7b262605968","modified":"2024-05-04T12:07:48.391828Z","created":"2024-05-04T12:07:48.391828Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi079","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:3300","nonce":0},{"type":"v1","addr":"172.21.15.79:6789","nonce":0}]},"addr":"172.21.15.79:6789/0","public_addr":"172.21.15.79:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-05-04T12:10:21.904 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:21.904+0000 7fc5bdd3c700 1 -- 172.21.15.146:0/450420403 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7fc5a00384b0 msgr2=0x7fc5a003a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:10:21.904 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:21.904+0000 7fc5bdd3c700 1 --2- 172.21.15.146:0/450420403 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7fc5a00384b0 0x7fc5a003a970 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fc5ac006fd0 tx=0x7fc5ac006e40 comp rx=0 tx=0).stop 2024-05-04T12:10:21.904 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:21.904+0000 7fc5bdd3c700 1 -- 172.21.15.146:0/450420403 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc5b80ff7d0 msgr2=0x7fc5b819c3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:10:21.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:21.904+0000 7fc5bdd3c700 1 --2- 172.21.15.146:0/450420403 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc5b80ff7d0 0x7fc5b819c3d0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fc5a8005b40 tx=0x7fc5a8004c80 comp rx=0 tx=0).stop 2024-05-04T12:10:21.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:21.905+0000 7fc5bdd3c700 1 -- 172.21.15.146:0/450420403 shutdown_connections 2024-05-04T12:10:21.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:21.905+0000 7fc5bdd3c700 1 --2- 172.21.15.146:0/450420403 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7fc5a00384b0 0x7fc5a003a970 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:10:21.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:21.905+0000 7fc5bdd3c700 1 --2- 172.21.15.146:0/450420403 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc5b80ff7d0 0x7fc5b819c3d0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:10:21.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:21.905+0000 7fc5bdd3c700 1 -- 172.21.15.146:0/450420403 >> 172.21.15.146:0/450420403 conn(0x7fc5b80fb250 msgr2=0x7fc5b80fbec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:10:21.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:21.905+0000 7fc5bdd3c700 1 -- 172.21.15.146:0/450420403 shutdown_connections 2024-05-04T12:10:21.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:21.905+0000 7fc5bdd3c700 1 -- 172.21.15.146:0/450420403 wait complete. 2024-05-04T12:10:21.906 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-05-04T12:10:23.195 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:22 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:10:23.196 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:22 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:10:23.196 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:22 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:10:23.196 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:22 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:10:23.196 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:22 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi146", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-05-04T12:10:23.196 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:22 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi146", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-05-04T12:10:23.196 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:22 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-04T12:10:23.197 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:22 smithi079 ceph-mon[27222]: Deploying daemon crash.smithi146 on smithi146 2024-05-04T12:10:23.197 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:22 smithi079 ceph-mon[27222]: from='client.? 172.21.15.146:0/450420403' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-04T12:10:23.498 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-05-04T12:10:23.498 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d9b985c4-0a0e-11ef-bc97-c7b262605968 -- ceph mon dump -f json 2024-05-04T12:10:23.808 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/d9b985c4-0a0e-11ef-bc97-c7b262605968/config/ceph.conf 2024-05-04T12:10:25.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:25.167+0000 7fc72ad53700 1 -- 172.21.15.146:0/900955261 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc724105720 msgr2=0x7fc724105b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:10:25.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:25.167+0000 7fc72ad53700 1 --2- 172.21.15.146:0/900955261 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc724105720 0x7fc724105b00 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fc718009910 tx=0x7fc718009c20 comp rx=0 tx=0).stop 2024-05-04T12:10:25.169 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:25.167+0000 7fc72ad53700 1 -- 172.21.15.146:0/900955261 shutdown_connections 2024-05-04T12:10:25.169 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:25.167+0000 7fc72ad53700 1 --2- 172.21.15.146:0/900955261 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc724105720 0x7fc724105b00 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:10:25.169 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:25.167+0000 7fc72ad53700 1 -- 172.21.15.146:0/900955261 >> 172.21.15.146:0/900955261 conn(0x7fc7240fb250 msgr2=0x7fc7240fd670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:10:25.169 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:25.167+0000 7fc72ad53700 1 -- 172.21.15.146:0/900955261 shutdown_connections 2024-05-04T12:10:25.169 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:25.167+0000 7fc72ad53700 1 -- 172.21.15.146:0/900955261 wait complete. 2024-05-04T12:10:25.169 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:25.167+0000 7fc72ad53700 1 Processor -- start 2024-05-04T12:10:25.169 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:25.167+0000 7fc72ad53700 1 -- start start 2024-05-04T12:10:25.169 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:25.168+0000 7fc72ad53700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc724105720 0x7fc7240fb700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:10:25.169 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:25.168+0000 7fc72ad53700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7240fd680 con 0x7fc724105720 2024-05-04T12:10:25.169 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:25.168+0000 7fc728aef700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc724105720 0x7fc7240fb700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:10:25.169 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:25.168+0000 7fc728aef700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc724105720 0x7fc7240fb700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.146:55646/0 (socket says 172.21.15.146:55646) 2024-05-04T12:10:25.169 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:25.168+0000 7fc728aef700 1 -- 172.21.15.146:0/3731431778 learned_addr learned my addr 172.21.15.146:0/3731431778 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-05-04T12:10:25.170 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:25.168+0000 7fc728aef700 1 -- 172.21.15.146:0/3731431778 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7180095f0 con 0x7fc724105720 2024-05-04T12:10:25.170 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:25.168+0000 7fc728aef700 1 --2- 172.21.15.146:0/3731431778 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc724105720 0x7fc7240fb700 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fc718005b40 tx=0x7fc718004c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:10:25.170 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:25.169+0000 7fc721ffb700 1 -- 172.21.15.146:0/3731431778 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc71801d070 con 0x7fc724105720 2024-05-04T12:10:25.170 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:25.169+0000 7fc72ad53700 1 -- 172.21.15.146:0/3731431778 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7240fbc40 con 0x7fc724105720 2024-05-04T12:10:25.170 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:25.169+0000 7fc721ffb700 1 -- 172.21.15.146:0/3731431778 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fc7180044d0 con 0x7fc724105720 2024-05-04T12:10:25.170 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:25.169+0000 7fc721ffb700 1 -- 172.21.15.146:0/3731431778 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc718003930 con 0x7fc724105720 2024-05-04T12:10:25.170 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:25.169+0000 7fc72ad53700 1 -- 172.21.15.146:0/3731431778 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7240fc0c0 con 0x7fc724105720 2024-05-04T12:10:25.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:25.170+0000 7fc72ad53700 1 -- 172.21.15.146:0/3731431778 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7240623c0 con 0x7fc724105720 2024-05-04T12:10:25.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:25.170+0000 7fc721ffb700 1 -- 172.21.15.146:0/3731431778 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 17) v1 ==== 45396+0+0 (secure 0 0 0) 0x7fc71800b960 con 0x7fc724105720 2024-05-04T12:10:25.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:25.170+0000 7fc721ffb700 1 --2- 172.21.15.146:0/3731431778 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7fc7140384b0 0x7fc71403a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:10:25.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:25.170+0000 7fc721ffb700 1 -- 172.21.15.146:0/3731431778 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7fc71804c5b0 con 0x7fc724105720 2024-05-04T12:10:25.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:25.170+0000 7fc723fff700 1 --2- 172.21.15.146:0/3731431778 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7fc7140384b0 0x7fc71403a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:10:25.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:25.171+0000 7fc723fff700 1 --2- 172.21.15.146:0/3731431778 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7fc7140384b0 0x7fc71403a970 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fc710006fd0 tx=0x7fc710006e40 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:10:25.174 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:25.175+0000 7fc721ffb700 1 -- 172.21.15.146:0/3731431778 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fc71802a540 con 0x7fc724105720 2024-05-04T12:10:25.456 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:25.456+0000 7fc72ad53700 1 -- 172.21.15.146:0/3731431778 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc7240fc550 con 0x7fc724105720 2024-05-04T12:10:25.456 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:25.456+0000 7fc721ffb700 1 -- 172.21.15.146:0/3731431778 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7fc71802a360 con 0x7fc724105720 2024-05-04T12:10:25.456 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-05-04T12:10:25.456 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"d9b985c4-0a0e-11ef-bc97-c7b262605968","modified":"2024-05-04T12:07:48.391828Z","created":"2024-05-04T12:07:48.391828Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi079","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:3300","nonce":0},{"type":"v1","addr":"172.21.15.79:6789","nonce":0}]},"addr":"172.21.15.79:6789/0","public_addr":"172.21.15.79:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-05-04T12:10:25.458 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:25.459+0000 7fc72ad53700 1 -- 172.21.15.146:0/3731431778 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7fc7140384b0 msgr2=0x7fc71403a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:10:25.458 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:25.459+0000 7fc72ad53700 1 --2- 172.21.15.146:0/3731431778 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7fc7140384b0 0x7fc71403a970 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fc710006fd0 tx=0x7fc710006e40 comp rx=0 tx=0).stop 2024-05-04T12:10:25.458 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:25.459+0000 7fc72ad53700 1 -- 172.21.15.146:0/3731431778 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc724105720 msgr2=0x7fc7240fb700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:10:25.458 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:25.459+0000 7fc72ad53700 1 --2- 172.21.15.146:0/3731431778 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc724105720 0x7fc7240fb700 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fc718005b40 tx=0x7fc718004c80 comp rx=0 tx=0).stop 2024-05-04T12:10:25.459 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:25.459+0000 7fc72ad53700 1 -- 172.21.15.146:0/3731431778 shutdown_connections 2024-05-04T12:10:25.459 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:25.459+0000 7fc72ad53700 1 --2- 172.21.15.146:0/3731431778 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7fc7140384b0 0x7fc71403a970 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:10:25.459 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:25.459+0000 7fc72ad53700 1 --2- 172.21.15.146:0/3731431778 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc724105720 0x7fc7240fb700 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:10:25.459 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:25.459+0000 7fc72ad53700 1 -- 172.21.15.146:0/3731431778 >> 172.21.15.146:0/3731431778 conn(0x7fc7240fb250 msgr2=0x7fc724107af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:10:25.459 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:25.459+0000 7fc72ad53700 1 -- 172.21.15.146:0/3731431778 shutdown_connections 2024-05-04T12:10:25.459 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:25.459+0000 7fc72ad53700 1 -- 172.21.15.146:0/3731431778 wait complete. 2024-05-04T12:10:25.459 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-05-04T12:10:25.945 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:25 smithi079 ceph-mon[27222]: from='client.? 172.21.15.146:0/3731431778' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-04T12:10:27.195 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:26 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:10:27.195 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:26 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:10:27.195 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:26 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:10:27.195 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:26 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:10:27.195 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:26 smithi079 ceph-mon[27222]: Deploying daemon node-exporter.smithi146 on smithi146 2024-05-04T12:10:27.195 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:26 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:10:27.195 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:26 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-04T12:10:27.270 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-05-04T12:10:27.270 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d9b985c4-0a0e-11ef-bc97-c7b262605968 -- ceph mon dump -f json 2024-05-04T12:10:27.472 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/d9b985c4-0a0e-11ef-bc97-c7b262605968/config/ceph.conf 2024-05-04T12:10:30.008 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:29.950+0000 7fc4532fb700 1 -- 172.21.15.146:0/1014423822 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc44c105780 msgr2=0x7fc44c105b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:10:30.008 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:29.950+0000 7fc4532fb700 1 --2- 172.21.15.146:0/1014423822 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc44c105780 0x7fc44c105b60 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fc43c009910 tx=0x7fc43c009c20 comp rx=0 tx=0).stop 2024-05-04T12:10:30.008 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:29.950+0000 7fc4532fb700 1 -- 172.21.15.146:0/1014423822 shutdown_connections 2024-05-04T12:10:30.008 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:29.950+0000 7fc4532fb700 1 --2- 172.21.15.146:0/1014423822 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc44c105780 0x7fc44c105b60 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:10:30.008 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:29.950+0000 7fc4532fb700 1 -- 172.21.15.146:0/1014423822 >> 172.21.15.146:0/1014423822 conn(0x7fc44c075290 msgr2=0x7fc44c0756a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:10:30.008 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:29.951+0000 7fc4532fb700 1 -- 172.21.15.146:0/1014423822 shutdown_connections 2024-05-04T12:10:30.008 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:29.951+0000 7fc4532fb700 1 -- 172.21.15.146:0/1014423822 wait complete. 2024-05-04T12:10:30.009 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:29.951+0000 7fc4532fb700 1 Processor -- start 2024-05-04T12:10:30.009 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:29.951+0000 7fc4532fb700 1 -- start start 2024-05-04T12:10:30.009 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:29.951+0000 7fc4532fb700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc44c105780 0x7fc44c194990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:10:30.009 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:29.951+0000 7fc4532fb700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc44c068e70 con 0x7fc44c105780 2024-05-04T12:10:30.009 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:29.951+0000 7fc451097700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc44c105780 0x7fc44c194990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:10:30.009 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:29.952+0000 7fc451097700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc44c105780 0x7fc44c194990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.146:55662/0 (socket says 172.21.15.146:55662) 2024-05-04T12:10:30.009 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:29.952+0000 7fc451097700 1 -- 172.21.15.146:0/234122362 learned_addr learned my addr 172.21.15.146:0/234122362 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-05-04T12:10:30.009 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:29.952+0000 7fc451097700 1 -- 172.21.15.146:0/234122362 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc43c0095f0 con 0x7fc44c105780 2024-05-04T12:10:30.009 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:29.952+0000 7fc451097700 1 --2- 172.21.15.146:0/234122362 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc44c105780 0x7fc44c194990 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fc43c005b40 tx=0x7fc43c004c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:10:30.009 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:29.952+0000 7fc4427fc700 1 -- 172.21.15.146:0/234122362 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc43c01d070 con 0x7fc44c105780 2024-05-04T12:10:30.009 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:29.952+0000 7fc4532fb700 1 -- 172.21.15.146:0/234122362 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc44c194ed0 con 0x7fc44c105780 2024-05-04T12:10:30.009 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:29.952+0000 7fc4427fc700 1 -- 172.21.15.146:0/234122362 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fc43c0044d0 con 0x7fc44c105780 2024-05-04T12:10:30.009 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:29.952+0000 7fc4532fb700 1 -- 172.21.15.146:0/234122362 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc44c195290 con 0x7fc44c105780 2024-05-04T12:10:30.010 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:29.952+0000 7fc4427fc700 1 -- 172.21.15.146:0/234122362 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc43c0039a0 con 0x7fc44c105780 2024-05-04T12:10:30.010 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:29.953+0000 7fc4532fb700 1 -- 172.21.15.146:0/234122362 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc44c1039c0 con 0x7fc44c105780 2024-05-04T12:10:30.010 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:29.953+0000 7fc4427fc700 1 -- 172.21.15.146:0/234122362 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 17) v1 ==== 45396+0+0 (secure 0 0 0) 0x7fc43c00b960 con 0x7fc44c105780 2024-05-04T12:10:30.010 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:29.953+0000 7fc4427fc700 1 --2- 172.21.15.146:0/234122362 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7fc4380384b0 0x7fc43803a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:10:30.010 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:29.953+0000 7fc4427fc700 1 -- 172.21.15.146:0/234122362 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7fc43c04c540 con 0x7fc44c105780 2024-05-04T12:10:30.010 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:29.953+0000 7fc450896700 1 --2- 172.21.15.146:0/234122362 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7fc4380384b0 0x7fc43803a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:10:30.010 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:29.954+0000 7fc450896700 1 --2- 172.21.15.146:0/234122362 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7fc4380384b0 0x7fc43803a970 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fc448006fd0 tx=0x7fc448006e40 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:10:30.010 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:29.958+0000 7fc4427fc700 1 -- 172.21.15.146:0/234122362 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fc43c02a540 con 0x7fc44c105780 2024-05-04T12:10:30.236 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:30.237+0000 7fc4532fb700 1 -- 172.21.15.146:0/234122362 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc44c195580 con 0x7fc44c105780 2024-05-04T12:10:30.236 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:30.237+0000 7fc4427fc700 1 -- 172.21.15.146:0/234122362 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7fc43c02a360 con 0x7fc44c105780 2024-05-04T12:10:30.237 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-05-04T12:10:30.237 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"d9b985c4-0a0e-11ef-bc97-c7b262605968","modified":"2024-05-04T12:07:48.391828Z","created":"2024-05-04T12:07:48.391828Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi079","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:3300","nonce":0},{"type":"v1","addr":"172.21.15.79:6789","nonce":0}]},"addr":"172.21.15.79:6789/0","public_addr":"172.21.15.79:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-05-04T12:10:30.239 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:30.239+0000 7fc4532fb700 1 -- 172.21.15.146:0/234122362 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7fc4380384b0 msgr2=0x7fc43803a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:10:30.239 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:30.239+0000 7fc4532fb700 1 --2- 172.21.15.146:0/234122362 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7fc4380384b0 0x7fc43803a970 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fc448006fd0 tx=0x7fc448006e40 comp rx=0 tx=0).stop 2024-05-04T12:10:30.239 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:30.239+0000 7fc4532fb700 1 -- 172.21.15.146:0/234122362 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc44c105780 msgr2=0x7fc44c194990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:10:30.239 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:30.239+0000 7fc4532fb700 1 --2- 172.21.15.146:0/234122362 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc44c105780 0x7fc44c194990 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fc43c005b40 tx=0x7fc43c004c80 comp rx=0 tx=0).stop 2024-05-04T12:10:30.239 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:30.239+0000 7fc4532fb700 1 -- 172.21.15.146:0/234122362 shutdown_connections 2024-05-04T12:10:30.239 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:30.239+0000 7fc4532fb700 1 --2- 172.21.15.146:0/234122362 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7fc4380384b0 0x7fc43803a970 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:10:30.239 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:30.239+0000 7fc4532fb700 1 --2- 172.21.15.146:0/234122362 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc44c105780 0x7fc44c194990 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:10:30.239 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:30.239+0000 7fc4532fb700 1 -- 172.21.15.146:0/234122362 >> 172.21.15.146:0/234122362 conn(0x7fc44c075290 msgr2=0x7fc44c0fefb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:10:30.239 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:30.239+0000 7fc4532fb700 1 -- 172.21.15.146:0/234122362 shutdown_connections 2024-05-04T12:10:30.240 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:30.240+0000 7fc4532fb700 1 -- 172.21.15.146:0/234122362 wait complete. 2024-05-04T12:10:30.240 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-05-04T12:10:30.695 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:30 smithi079 ceph-mon[27222]: from='client.? 172.21.15.146:0/234122362' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-04T12:10:31.945 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:31 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:10:31.945 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:31 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:10:31.945 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:31 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:10:31.945 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:31 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:10:31.945 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:31 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi146.eubqih", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-04T12:10:31.945 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:31 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.smithi146.eubqih", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-05-04T12:10:31.945 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:31 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-04T12:10:31.945 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:31 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-04T12:10:31.946 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:31 smithi079 ceph-mon[27222]: Deploying daemon mgr.smithi146.eubqih on smithi146 2024-05-04T12:10:32.020 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-05-04T12:10:32.020 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d9b985c4-0a0e-11ef-bc97-c7b262605968 -- ceph mon dump -f json 2024-05-04T12:10:32.229 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/d9b985c4-0a0e-11ef-bc97-c7b262605968/config/ceph.conf 2024-05-04T12:10:32.868 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:32 smithi079 ceph-mon[27222]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-04T12:10:32.868 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:32 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:10:34.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:34.185+0000 7f0d30345700 1 -- 172.21.15.146:0/121101094 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0d28106820 msgr2=0x7f0d28106c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:10:34.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:34.185+0000 7f0d30345700 1 --2- 172.21.15.146:0/121101094 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0d28106820 0x7f0d28106c00 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f0d24009910 tx=0x7f0d24009c20 comp rx=0 tx=0).stop 2024-05-04T12:10:34.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:34.185+0000 7f0d30345700 1 -- 172.21.15.146:0/121101094 shutdown_connections 2024-05-04T12:10:34.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:34.185+0000 7f0d30345700 1 --2- 172.21.15.146:0/121101094 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0d28106820 0x7f0d28106c00 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:10:34.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:34.185+0000 7f0d30345700 1 -- 172.21.15.146:0/121101094 >> 172.21.15.146:0/121101094 conn(0x7f0d28075290 msgr2=0x7f0d280756a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:10:34.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:34.185+0000 7f0d30345700 1 -- 172.21.15.146:0/121101094 shutdown_connections 2024-05-04T12:10:34.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:34.185+0000 7f0d30345700 1 -- 172.21.15.146:0/121101094 wait complete. 2024-05-04T12:10:34.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:34.186+0000 7f0d30345700 1 Processor -- start 2024-05-04T12:10:34.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:34.186+0000 7f0d30345700 1 -- start start 2024-05-04T12:10:34.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:34.186+0000 7f0d30345700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0d28106820 0x7f0d2819bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:10:34.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:34.186+0000 7f0d30345700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d2819c060 con 0x7f0d28106820 2024-05-04T12:10:34.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:34.186+0000 7f0d2e0e1700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0d28106820 0x7f0d2819bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:10:34.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:34.186+0000 7f0d2e0e1700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0d28106820 0x7f0d2819bb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.146:55668/0 (socket says 172.21.15.146:55668) 2024-05-04T12:10:34.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:34.186+0000 7f0d2e0e1700 1 -- 172.21.15.146:0/2946378486 learned_addr learned my addr 172.21.15.146:0/2946378486 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-05-04T12:10:34.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:34.186+0000 7f0d2e0e1700 1 -- 172.21.15.146:0/2946378486 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d240095f0 con 0x7f0d28106820 2024-05-04T12:10:34.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:34.187+0000 7f0d2e0e1700 1 --2- 172.21.15.146:0/2946378486 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0d28106820 0x7f0d2819bb20 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f0d24005b40 tx=0x7f0d24004c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:10:34.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:34.187+0000 7f0d1f7fe700 1 -- 172.21.15.146:0/2946378486 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0d2401d070 con 0x7f0d28106820 2024-05-04T12:10:34.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:34.187+0000 7f0d1f7fe700 1 -- 172.21.15.146:0/2946378486 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f0d240044d0 con 0x7f0d28106820 2024-05-04T12:10:34.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:34.187+0000 7f0d30345700 1 -- 172.21.15.146:0/2946378486 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d2819c260 con 0x7f0d28106820 2024-05-04T12:10:34.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:34.187+0000 7f0d1f7fe700 1 -- 172.21.15.146:0/2946378486 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0d24003910 con 0x7f0d28106820 2024-05-04T12:10:34.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:34.187+0000 7f0d30345700 1 -- 172.21.15.146:0/2946378486 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d28195b10 con 0x7f0d28106820 2024-05-04T12:10:34.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:34.188+0000 7f0d1f7fe700 1 -- 172.21.15.146:0/2946378486 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 17) v1 ==== 45396+0+0 (secure 0 0 0) 0x7f0d24003b90 con 0x7f0d28106820 2024-05-04T12:10:34.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:34.188+0000 7f0d30345700 1 -- 172.21.15.146:0/2946378486 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d280623c0 con 0x7f0d28106820 2024-05-04T12:10:34.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:34.188+0000 7f0d1f7fe700 1 --2- 172.21.15.146:0/2946378486 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f0d140384b0 0x7f0d1403a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:10:34.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:34.188+0000 7f0d1f7fe700 1 -- 172.21.15.146:0/2946378486 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7f0d2404c460 con 0x7f0d28106820 2024-05-04T12:10:34.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:34.188+0000 7f0d2d8e0700 1 --2- 172.21.15.146:0/2946378486 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f0d140384b0 0x7f0d1403a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:10:34.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:34.189+0000 7f0d2d8e0700 1 --2- 172.21.15.146:0/2946378486 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f0d140384b0 0x7f0d1403a970 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f0d18006fd0 tx=0x7f0d18006e40 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:10:34.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:34.193+0000 7f0d1f7fe700 1 -- 172.21.15.146:0/2946378486 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f0d2402a540 con 0x7f0d28106820 2024-05-04T12:10:34.484 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:34.485+0000 7f0d30345700 1 -- 172.21.15.146:0/2946378486 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0d281965a0 con 0x7f0d28106820 2024-05-04T12:10:34.485 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:34.485+0000 7f0d1f7fe700 1 -- 172.21.15.146:0/2946378486 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f0d2402a360 con 0x7f0d28106820 2024-05-04T12:10:34.485 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-05-04T12:10:34.485 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"d9b985c4-0a0e-11ef-bc97-c7b262605968","modified":"2024-05-04T12:07:48.391828Z","created":"2024-05-04T12:07:48.391828Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi079","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:3300","nonce":0},{"type":"v1","addr":"172.21.15.79:6789","nonce":0}]},"addr":"172.21.15.79:6789/0","public_addr":"172.21.15.79:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-05-04T12:10:34.487 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:34.488+0000 7f0d30345700 1 -- 172.21.15.146:0/2946378486 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f0d140384b0 msgr2=0x7f0d1403a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:10:34.487 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:34.488+0000 7f0d30345700 1 --2- 172.21.15.146:0/2946378486 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f0d140384b0 0x7f0d1403a970 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f0d18006fd0 tx=0x7f0d18006e40 comp rx=0 tx=0).stop 2024-05-04T12:10:34.487 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:34.488+0000 7f0d30345700 1 -- 172.21.15.146:0/2946378486 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0d28106820 msgr2=0x7f0d2819bb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:10:34.487 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:34.488+0000 7f0d30345700 1 --2- 172.21.15.146:0/2946378486 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0d28106820 0x7f0d2819bb20 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f0d24005b40 tx=0x7f0d24004c80 comp rx=0 tx=0).stop 2024-05-04T12:10:34.488 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:34.488+0000 7f0d30345700 1 -- 172.21.15.146:0/2946378486 shutdown_connections 2024-05-04T12:10:34.488 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:34.488+0000 7f0d30345700 1 --2- 172.21.15.146:0/2946378486 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f0d140384b0 0x7f0d1403a970 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:10:34.488 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:34.488+0000 7f0d30345700 1 --2- 172.21.15.146:0/2946378486 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0d28106820 0x7f0d2819bb20 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:10:34.488 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:34.488+0000 7f0d30345700 1 -- 172.21.15.146:0/2946378486 >> 172.21.15.146:0/2946378486 conn(0x7f0d28075290 msgr2=0x7f0d280fd4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:10:34.488 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:34.488+0000 7f0d30345700 1 -- 172.21.15.146:0/2946378486 shutdown_connections 2024-05-04T12:10:34.488 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:34.488+0000 7f0d30345700 1 -- 172.21.15.146:0/2946378486 wait complete. 2024-05-04T12:10:34.488 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-05-04T12:10:34.945 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:34 smithi079 ceph-mon[27222]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-04T12:10:34.945 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:34 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:10:34.945 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:34 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:10:34.946 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:34 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:10:34.946 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:34 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:10:34.946 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:34 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-04T12:10:34.946 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:34 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-04T12:10:34.946 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:34 smithi079 ceph-mon[27222]: from='client.? 172.21.15.146:0/2946378486' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-04T12:10:35.945 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:35 smithi079 ceph-mon[27222]: Deploying daemon mon.smithi146 on smithi146 2024-05-04T12:10:36.078 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-05-04T12:10:36.079 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d9b985c4-0a0e-11ef-bc97-c7b262605968 -- ceph mon dump -f json 2024-05-04T12:10:36.925 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/d9b985c4-0a0e-11ef-bc97-c7b262605968/config/ceph.conf 2024-05-04T12:10:37.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:36 smithi079 ceph-mon[27222]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-04T12:10:37.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:36 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:10:38.386 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:38 smithi146 systemd[1]: Starting Ceph mon.smithi146 for d9b985c4-0a0e-11ef-bc97-c7b262605968... 2024-05-04T12:10:39.251 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 podman[35264]: 2024-05-04 12:10:38.928873714 +0000 UTC m=+0.072286200 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3 2024-05-04T12:10:39.251 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 podman[35264]: 2024-05-04 12:10:39.057588287 +0000 UTC m=+0.201000757 container create 085b6b3257eecaf5105784eeb8c64a3be9070d608f6dffc215de00b5b5d2e2a4 (image=quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3, name=ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi146, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=HEAD, io.buildah.version=1.29.1, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.1, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git) 2024-05-04T12:10:39.349 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:39.263+0000 7f67f5190700 1 -- 172.21.15.146:0/1339309619 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f67f0105700 msgr2=0x7f67f0105ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:10:39.349 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:39.263+0000 7f67f5190700 1 --2- 172.21.15.146:0/1339309619 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f67f0105700 0x7f67f0105ae0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f67e0009910 tx=0x7f67e0009c20 comp rx=0 tx=0).stop 2024-05-04T12:10:39.349 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:39.263+0000 7f67f5190700 1 -- 172.21.15.146:0/1339309619 shutdown_connections 2024-05-04T12:10:39.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:39.263+0000 7f67f5190700 1 --2- 172.21.15.146:0/1339309619 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f67f0105700 0x7f67f0105ae0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:10:39.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:39.263+0000 7f67f5190700 1 -- 172.21.15.146:0/1339309619 >> 172.21.15.146:0/1339309619 conn(0x7f67f00fb210 msgr2=0x7f67f00fd630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:10:39.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:39.263+0000 7f67f5190700 1 -- 172.21.15.146:0/1339309619 shutdown_connections 2024-05-04T12:10:39.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:39.263+0000 7f67f5190700 1 -- 172.21.15.146:0/1339309619 wait complete. 2024-05-04T12:10:39.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:39.263+0000 7f67f5190700 1 Processor -- start 2024-05-04T12:10:39.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:39.264+0000 7f67f5190700 1 -- start start 2024-05-04T12:10:39.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:39.264+0000 7f67f5190700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f67f0105700 0x7f67f0196b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:10:39.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:39.264+0000 7f67f5190700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67f0197080 con 0x7f67f0105700 2024-05-04T12:10:39.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:39.264+0000 7f67eed9d700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f67f0105700 0x7f67f0196b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:10:39.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:39.264+0000 7f67eed9d700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f67f0105700 0x7f67f0196b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.146:58078/0 (socket says 172.21.15.146:58078) 2024-05-04T12:10:39.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:39.264+0000 7f67eed9d700 1 -- 172.21.15.146:0/1968353948 learned_addr learned my addr 172.21.15.146:0/1968353948 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-05-04T12:10:39.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:39.265+0000 7f67eed9d700 1 -- 172.21.15.146:0/1968353948 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67e00095f0 con 0x7f67f0105700 2024-05-04T12:10:39.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:39.265+0000 7f67eed9d700 1 --2- 172.21.15.146:0/1968353948 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f67f0105700 0x7f67f0196b40 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f67e0005b40 tx=0x7f67e0004c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:10:39.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:39.265+0000 7f67d7fff700 1 -- 172.21.15.146:0/1968353948 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f67e001d070 con 0x7f67f0105700 2024-05-04T12:10:39.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:39.265+0000 7f67f5190700 1 -- 172.21.15.146:0/1968353948 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67f0197280 con 0x7f67f0105700 2024-05-04T12:10:39.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:39.265+0000 7f67d7fff700 1 -- 172.21.15.146:0/1968353948 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f67e00044d0 con 0x7f67f0105700 2024-05-04T12:10:39.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:39.265+0000 7f67f5190700 1 -- 172.21.15.146:0/1968353948 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67f0197700 con 0x7f67f0105700 2024-05-04T12:10:39.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:39.265+0000 7f67d7fff700 1 -- 172.21.15.146:0/1968353948 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f67e0003910 con 0x7f67f0105700 2024-05-04T12:10:39.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:39.266+0000 7f67f5190700 1 -- 172.21.15.146:0/1968353948 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67f01039b0 con 0x7f67f0105700 2024-05-04T12:10:39.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:39.266+0000 7f67d7fff700 1 -- 172.21.15.146:0/1968353948 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 17) v1 ==== 45396+0+0 (secure 0 0 0) 0x7f67e0003bb0 con 0x7f67f0105700 2024-05-04T12:10:39.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:39.266+0000 7f67d7fff700 1 --2- 172.21.15.146:0/1968353948 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f67d80384b0 0x7f67d803a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:10:39.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:39.266+0000 7f67d7fff700 1 -- 172.21.15.146:0/1968353948 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7f67e004c560 con 0x7f67f0105700 2024-05-04T12:10:39.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:39.266+0000 7f67ee59c700 1 --2- 172.21.15.146:0/1968353948 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f67d80384b0 0x7f67d803a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:10:39.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:39.267+0000 7f67ee59c700 1 --2- 172.21.15.146:0/1968353948 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f67d80384b0 0x7f67d803a970 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f67e4006fd0 tx=0x7f67e4006e40 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:10:39.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:39.271+0000 7f67d7fff700 1 -- 172.21.15.146:0/1968353948 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f67e002a540 con 0x7f67f0105700 2024-05-04T12:10:39.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:38 smithi079 ceph-mon[27222]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-04T12:10:39.494 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:39.495+0000 7f67d7fff700 1 -- 172.21.15.146:0/1968353948 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f67e002a540 con 0x7f67f0105700 2024-05-04T12:10:39.517 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 podman[35264]: 2024-05-04 12:10:39.435847438 +0000 UTC m=+0.579259918 container init 085b6b3257eecaf5105784eeb8c64a3be9070d608f6dffc215de00b5b5d2e2a4 (image=quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3, name=ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi146, io.buildah.version=1.29.1, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=-18.2.1, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-05-04T12:10:39.517 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 podman[35264]: 2024-05-04 12:10:39.438928138 +0000 UTC m=+0.582340611 container start 085b6b3257eecaf5105784eeb8c64a3be9070d608f6dffc215de00b5b5d2e2a4 (image=quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3, name=ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi146, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-05-04T12:10:39.517 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: set uid:gid to 167:167 (ceph:ceph) 2024-05-04T12:10:39.517 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable), process ceph-mon, pid 2 2024-05-04T12:10:39.517 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: pidfile_write: ignore empty --pid-file 2024-05-04T12:10:39.517 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: load: jerasure load: lrc 2024-05-04T12:10:39.518 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: RocksDB version: 7.9.2 2024-05-04T12:10:39.518 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Git sha 0 2024-05-04T12:10:39.518 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Compile date 2023-12-11 22:07:34 2024-05-04T12:10:39.518 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: DB SUMMARY 2024-05-04T12:10:39.518 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: DB Session ID: J2LXB1MI7A7YWEKNA77M 2024-05-04T12:10:39.518 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: CURRENT file: CURRENT 2024-05-04T12:10:39.518 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: IDENTITY file: IDENTITY 2024-05-04T12:10:39.518 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-05-04T12:10:39.518 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi146/store.db dir, Total Num: 0, files: 2024-05-04T12:10:39.518 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi146/store.db: 000004.log size: 511 ; 2024-05-04T12:10:39.518 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.error_if_exists: 0 2024-05-04T12:10:39.519 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.create_if_missing: 0 2024-05-04T12:10:39.519 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.paranoid_checks: 1 2024-05-04T12:10:39.519 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.flush_verify_memtable_count: 1 2024-05-04T12:10:39.519 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-05-04T12:10:39.519 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-05-04T12:10:39.519 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.env: 0x560d7cdd8720 2024-05-04T12:10:39.519 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.fs: PosixFileSystem 2024-05-04T12:10:39.519 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.info_log: 0x560d7f45f2e0 2024-05-04T12:10:39.519 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.max_file_opening_threads: 16 2024-05-04T12:10:39.519 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.statistics: (nil) 2024-05-04T12:10:39.521 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.use_fsync: 0 2024-05-04T12:10:39.521 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.max_log_file_size: 0 2024-05-04T12:10:39.521 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-05-04T12:10:39.521 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.log_file_time_to_roll: 0 2024-05-04T12:10:39.521 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.keep_log_file_num: 1000 2024-05-04T12:10:39.521 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.recycle_log_file_num: 0 2024-05-04T12:10:39.521 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.allow_fallocate: 1 2024-05-04T12:10:39.521 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.allow_mmap_reads: 0 2024-05-04T12:10:39.521 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.allow_mmap_writes: 0 2024-05-04T12:10:39.522 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.use_direct_reads: 0 2024-05-04T12:10:39.522 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-05-04T12:10:39.522 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.create_missing_column_families: 0 2024-05-04T12:10:39.522 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.db_log_dir: 2024-05-04T12:10:39.522 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.wal_dir: 2024-05-04T12:10:39.522 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.table_cache_numshardbits: 6 2024-05-04T12:10:39.522 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.WAL_ttl_seconds: 0 2024-05-04T12:10:39.522 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.WAL_size_limit_MB: 0 2024-05-04T12:10:39.522 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-05-04T12:10:39.522 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-05-04T12:10:39.522 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.is_fd_close_on_exec: 1 2024-05-04T12:10:39.522 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.advise_random_on_open: 1 2024-05-04T12:10:39.522 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.db_write_buffer_size: 0 2024-05-04T12:10:39.523 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.write_buffer_manager: 0x560d7e6d0140 2024-05-04T12:10:39.523 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-05-04T12:10:39.523 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-05-04T12:10:39.523 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.use_adaptive_mutex: 0 2024-05-04T12:10:39.523 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.rate_limiter: (nil) 2024-05-04T12:10:39.523 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-05-04T12:10:39.523 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.wal_recovery_mode: 2 2024-05-04T12:10:39.523 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.enable_thread_tracking: 0 2024-05-04T12:10:39.523 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.enable_pipelined_write: 0 2024-05-04T12:10:39.523 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.unordered_write: 0 2024-05-04T12:10:39.523 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-05-04T12:10:39.523 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-05-04T12:10:39.523 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-05-04T12:10:39.523 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-05-04T12:10:39.523 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.row_cache: None 2024-05-04T12:10:39.523 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.wal_filter: None 2024-05-04T12:10:39.524 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-05-04T12:10:39.524 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.allow_ingest_behind: 0 2024-05-04T12:10:39.524 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.two_write_queues: 0 2024-05-04T12:10:39.524 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.manual_wal_flush: 0 2024-05-04T12:10:39.524 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.wal_compression: 0 2024-05-04T12:10:39.524 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.atomic_flush: 0 2024-05-04T12:10:39.524 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-05-04T12:10:39.524 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.persist_stats_to_disk: 0 2024-05-04T12:10:39.524 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.write_dbid_to_manifest: 0 2024-05-04T12:10:39.524 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.log_readahead_size: 0 2024-05-04T12:10:39.524 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-05-04T12:10:39.524 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.best_efforts_recovery: 0 2024-05-04T12:10:39.524 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-05-04T12:10:39.524 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-05-04T12:10:39.524 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.allow_data_in_errors: 0 2024-05-04T12:10:39.525 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.db_host_id: __hostname__ 2024-05-04T12:10:39.525 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.enforce_single_del_contracts: true 2024-05-04T12:10:39.525 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.max_background_jobs: 2 2024-05-04T12:10:39.525 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.max_background_compactions: -1 2024-05-04T12:10:39.525 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.max_subcompactions: 1 2024-05-04T12:10:39.525 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-05-04T12:10:39.525 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-05-04T12:10:39.525 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.delayed_write_rate : 16777216 2024-05-04T12:10:39.525 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.max_total_wal_size: 0 2024-05-04T12:10:39.525 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-05-04T12:10:39.525 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.stats_dump_period_sec: 600 2024-05-04T12:10:39.525 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.stats_persist_period_sec: 600 2024-05-04T12:10:39.525 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-05-04T12:10:39.525 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.max_open_files: -1 2024-05-04T12:10:39.525 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.bytes_per_sync: 0 2024-05-04T12:10:39.526 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.wal_bytes_per_sync: 0 2024-05-04T12:10:39.526 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.strict_bytes_per_sync: 0 2024-05-04T12:10:39.526 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.compaction_readahead_size: 0 2024-05-04T12:10:39.526 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.max_background_flushes: -1 2024-05-04T12:10:39.526 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Compression algorithms supported: 2024-05-04T12:10:39.526 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-05-04T12:10:39.526 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: kZSTD supported: 0 2024-05-04T12:10:39.526 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: kXpressCompression supported: 0 2024-05-04T12:10:39.526 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: kLZ4HCCompression supported: 1 2024-05-04T12:10:39.526 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: kZlibCompression supported: 1 2024-05-04T12:10:39.526 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: kSnappyCompression supported: 1 2024-05-04T12:10:39.526 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: kLZ4Compression supported: 1 2024-05-04T12:10:39.526 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: kBZip2Compression supported: 0 2024-05-04T12:10:39.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Fast CRC32 supported: Supported on x86 2024-05-04T12:10:39.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: DMutex implementation: pthread_mutex_t 2024-05-04T12:10:39.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi146/store.db/MANIFEST-000005 2024-05-04T12:10:39.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-05-04T12:10:39.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-05-04T12:10:39.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.merge_operator: 2024-05-04T12:10:39.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.compaction_filter: None 2024-05-04T12:10:39.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.compaction_filter_factory: None 2024-05-04T12:10:39.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.sst_partitioner_factory: None 2024-05-04T12:10:39.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.memtable_factory: SkipListFactory 2024-05-04T12:10:39.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.table_factory: BlockBasedTable 2024-05-04T12:10:39.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x560d7f45f3e0) 2024-05-04T12:10:39.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: cache_index_and_filter_blocks: 1 2024-05-04T12:10:39.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-05-04T12:10:39.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-05-04T12:10:39.528 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: pin_top_level_index_and_filter: 1 2024-05-04T12:10:39.528 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: index_type: 0 2024-05-04T12:10:39.528 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: data_block_index_type: 0 2024-05-04T12:10:39.528 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: index_shortening: 1 2024-05-04T12:10:39.528 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: data_block_hash_table_util_ratio: 0.750000 2024-05-04T12:10:39.528 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: checksum: 4 2024-05-04T12:10:39.528 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: no_block_cache: 0 2024-05-04T12:10:39.528 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: block_cache: 0x560d7e771350 2024-05-04T12:10:39.528 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: block_cache_name: BinnedLRUCache 2024-05-04T12:10:39.528 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: block_cache_options: 2024-05-04T12:10:39.528 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: capacity : 536870912 2024-05-04T12:10:39.528 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: num_shard_bits : 4 2024-05-04T12:10:39.528 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: strict_capacity_limit : 0 2024-05-04T12:10:39.528 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: high_pri_pool_ratio: 0.000 2024-05-04T12:10:39.529 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: block_cache_compressed: (nil) 2024-05-04T12:10:39.529 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: persistent_cache: (nil) 2024-05-04T12:10:39.529 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: block_size: 4096 2024-05-04T12:10:39.529 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: block_size_deviation: 10 2024-05-04T12:10:39.529 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: block_restart_interval: 16 2024-05-04T12:10:39.529 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: index_block_restart_interval: 1 2024-05-04T12:10:39.529 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: metadata_block_size: 4096 2024-05-04T12:10:39.529 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: partition_filters: 0 2024-05-04T12:10:39.529 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: use_delta_encoding: 1 2024-05-04T12:10:39.529 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: filter_policy: bloomfilter 2024-05-04T12:10:39.529 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: whole_key_filtering: 1 2024-05-04T12:10:39.529 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: verify_compression: 0 2024-05-04T12:10:39.529 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: read_amp_bytes_per_bit: 0 2024-05-04T12:10:39.529 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: format_version: 5 2024-05-04T12:10:39.530 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: enable_index_compression: 1 2024-05-04T12:10:39.530 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: block_align: 0 2024-05-04T12:10:39.530 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: max_auto_readahead_size: 262144 2024-05-04T12:10:39.530 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: prepopulate_block_cache: 0 2024-05-04T12:10:39.530 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: initial_auto_readahead_size: 8192 2024-05-04T12:10:39.530 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: num_file_reads_for_auto_readahead: 2 2024-05-04T12:10:39.530 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.write_buffer_size: 33554432 2024-05-04T12:10:39.530 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.max_write_buffer_number: 2 2024-05-04T12:10:39.530 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.compression: NoCompression 2024-05-04T12:10:39.530 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.bottommost_compression: Disabled 2024-05-04T12:10:39.530 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.prefix_extractor: nullptr 2024-05-04T12:10:39.530 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-05-04T12:10:39.530 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.num_levels: 7 2024-05-04T12:10:39.530 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-05-04T12:10:39.531 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-05-04T12:10:39.531 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-05-04T12:10:39.531 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-05-04T12:10:39.531 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-05-04T12:10:39.531 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-05-04T12:10:39.531 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-05-04T12:10:39.531 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-05-04T12:10:39.531 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-05-04T12:10:39.531 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-05-04T12:10:39.531 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-05-04T12:10:39.531 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-05-04T12:10:39.531 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.compression_opts.window_bits: -14 2024-05-04T12:10:39.531 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.compression_opts.level: 32767 2024-05-04T12:10:39.531 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.compression_opts.strategy: 0 2024-05-04T12:10:39.532 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-05-04T12:10:39.532 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-05-04T12:10:39.532 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-05-04T12:10:39.532 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-05-04T12:10:39.532 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.compression_opts.enabled: false 2024-05-04T12:10:39.532 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-05-04T12:10:39.532 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-05-04T12:10:39.532 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-05-04T12:10:39.532 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-05-04T12:10:39.532 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.target_file_size_base: 67108864 2024-05-04T12:10:39.532 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.target_file_size_multiplier: 1 2024-05-04T12:10:39.532 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-05-04T12:10:39.532 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-05-04T12:10:39.532 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-05-04T12:10:39.532 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-05-04T12:10:39.532 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-05-04T12:10:39.533 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-05-04T12:10:39.533 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-05-04T12:10:39.533 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-05-04T12:10:39.533 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-05-04T12:10:39.533 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-05-04T12:10:39.533 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-05-04T12:10:39.533 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-05-04T12:10:39.533 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-05-04T12:10:39.533 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.arena_block_size: 1048576 2024-05-04T12:10:39.533 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-05-04T12:10:39.533 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-05-04T12:10:39.533 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.disable_auto_compactions: 0 2024-05-04T12:10:39.533 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-05-04T12:10:39.534 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-05-04T12:10:39.534 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-05-04T12:10:39.534 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-05-04T12:10:39.534 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-05-04T12:10:39.534 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-05-04T12:10:39.534 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-05-04T12:10:39.534 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-05-04T12:10:39.534 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-05-04T12:10:39.534 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-05-04T12:10:39.534 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.table_properties_collectors: 2024-05-04T12:10:39.534 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.inplace_update_support: 0 2024-05-04T12:10:39.534 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.inplace_update_num_locks: 10000 2024-05-04T12:10:39.534 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-05-04T12:10:39.535 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-05-04T12:10:39.535 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.memtable_huge_page_size: 0 2024-05-04T12:10:39.535 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.bloom_locality: 0 2024-05-04T12:10:39.535 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.max_successive_merges: 0 2024-05-04T12:10:39.535 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.optimize_filters_for_hits: 0 2024-05-04T12:10:39.535 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.paranoid_file_checks: 0 2024-05-04T12:10:39.535 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.force_consistency_checks: 1 2024-05-04T12:10:39.535 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.report_bg_io_stats: 0 2024-05-04T12:10:39.535 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.ttl: 2592000 2024-05-04T12:10:39.535 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.periodic_compaction_seconds: 0 2024-05-04T12:10:39.535 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-05-04T12:10:39.535 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-05-04T12:10:39.535 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.enable_blob_files: false 2024-05-04T12:10:39.535 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.min_blob_size: 0 2024-05-04T12:10:39.535 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.blob_file_size: 268435456 2024-05-04T12:10:39.536 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.blob_compression_type: NoCompression 2024-05-04T12:10:39.536 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.enable_blob_garbage_collection: false 2024-05-04T12:10:39.536 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-05-04T12:10:39.536 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-05-04T12:10:39.536 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-05-04T12:10:39.536 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.blob_file_starting_level: 0 2024-05-04T12:10:39.536 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-05-04T12:10:39.536 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi146/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-05-04T12:10:39.536 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-05-04T12:10:39.536 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: b3b8167b-48e0-467b-a129-3c87f8274cea 2024-05-04T12:10:39.536 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714824639467542, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-05-04T12:10:39.536 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-05-04T12:10:39.536 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714824639467966, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1622, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1714824639, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "b3b8167b-48e0-467b-a129-3c87f8274cea", "db_session_id": "J2LXB1MI7A7YWEKNA77M", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-05-04T12:10:39.536 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714824639468045, "job": 1, "event": "recovery_finished"} 2024-05-04T12:10:39.537 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-05-04T12:10:39.537 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-smithi146/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-05-04T12:10:39.537 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x560d7e80e000 2024-05-04T12:10:39.537 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: DB pointer 0x560d7e7fa000 2024-05-04T12:10:39.537 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-05-04T12:10:39.537 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-05-04T12:10:39.537 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: ** DB Stats ** 2024-05-04T12:10:39.537 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-05-04T12:10:39.537 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-05-04T12:10:39.537 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-05-04T12:10:39.537 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-05-04T12:10:39.537 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-05-04T12:10:39.537 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-05-04T12:10:39.537 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-05-04T12:10:39.537 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: 2024-05-04T12:10:39.538 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: ** Compaction Stats [default] ** 2024-05-04T12:10:39.538 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-05-04T12:10:39.538 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-05-04T12:10:39.538 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: L0 1/0 1.58 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.8 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-05-04T12:10:39.538 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: Sum 1/0 1.58 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.8 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-05-04T12:10:39.538 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.8 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-05-04T12:10:39.538 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: 2024-05-04T12:10:39.538 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: ** Compaction Stats [default] ** 2024-05-04T12:10:39.538 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-05-04T12:10:39.538 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-05-04T12:10:39.538 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.8 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-05-04T12:10:39.538 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: 2024-05-04T12:10:39.538 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-05-04T12:10:39.538 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: 2024-05-04T12:10:39.539 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-05-04T12:10:39.539 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-05-04T12:10:39.539 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-05-04T12:10:39.539 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-05-04T12:10:39.539 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-05-04T12:10:39.539 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: AddFile(Keys): cumulative 0, interval 0 2024-05-04T12:10:39.539 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: Cumulative compaction: 0.00 GB write, 0.64 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-04T12:10:39.539 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: Interval compaction: 0.00 GB write, 0.64 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-04T12:10:39.539 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-05-04T12:10:39.539 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: Block cache BinnedLRUCache@0x560d7e771350#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-05-04T12:10:39.539 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-05-04T12:10:39.539 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: 2024-05-04T12:10:39.539 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: ** File Read Latency Histogram By Level [default] ** 2024-05-04T12:10:39.540 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: mon.smithi146 does not exist in monmap, will attempt to join an existing cluster 2024-05-04T12:10:39.540 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: using public_addr v2:172.21.15.146:0/0 -> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] 2024-05-04T12:10:39.540 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: starting mon.smithi146 rank -1 at public addrs [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] at bind addrs [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi146 fsid d9b985c4-0a0e-11ef-bc97-c7b262605968 2024-05-04T12:10:39.540 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: mon.smithi146@-1(???) e0 preinit fsid d9b985c4-0a0e-11ef-bc97-c7b262605968 2024-05-04T12:10:39.540 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: mon.smithi146@-1(synchronizing).mds e1 new map 2024-05-04T12:10:39.540 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: mon.smithi146@-1(synchronizing).mds e1 print_map 2024-05-04T12:10:39.540 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: e1 2024-05-04T12:10:39.540 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-05-04T12:10:39.540 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-05-04T12:10:39.540 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: legacy client fscid: -1 2024-05-04T12:10:39.540 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: 2024-05-04T12:10:39.540 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout: No filesystems configured 2024-05-04T12:10:39.540 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: mon.smithi146@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-05-04T12:10:39.540 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: mon.smithi146@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-05-04T12:10:39.540 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: mon.smithi146@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-05-04T12:10:39.541 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: mon.smithi146@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-05-04T12:10:39.541 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: mon.smithi146@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-05-04T12:10:39.541 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: mon.smithi146@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-05-04T12:10:39.541 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: mon.smithi146@-1(synchronizing).osd e5 e5: 0 total, 0 up, 0 in 2024-05-04T12:10:39.541 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: mon.smithi146@-1(synchronizing).osd e5 crush map has features 3314932999778484224, adjusting msgr requires 2024-05-04T12:10:39.541 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: mon.smithi146@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2024-05-04T12:10:39.541 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: mon.smithi146@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2024-05-04T12:10:39.541 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: mon.smithi146@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2024-05-04T12:10:39.541 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-04T12:10:39.541 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:10:39.541 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:10:39.541 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:10:39.541 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:10:39.541 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-04T12:10:39.541 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-04T12:10:39.542 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: from='client.? 172.21.15.146:0/2946378486' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-04T12:10:39.542 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: Deploying daemon mon.smithi146 on smithi146 2024-05-04T12:10:39.542 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-04T12:10:39.542 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:10:39.542 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-04T12:10:39.542 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: mon.smithi146@-1(synchronizing).paxosservice(auth 1..8) refresh upgraded, format 0 -> 3 2024-05-04T12:10:39.542 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: mon.smithi146@-1(synchronizing) e1 handle_conf_change mon_allow_pool_delete,mon_cluster_log_to_file 2024-05-04T12:10:39.542 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: expand_channel_meta expand map: {default=false} 2024-05-04T12:10:39.542 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: expand_channel_meta from 'false' to 'false' 2024-05-04T12:10:39.542 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: expand_channel_meta expanded map: {default=false} 2024-05-04T12:10:39.542 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: expand_channel_meta expand map: {default=info} 2024-05-04T12:10:39.542 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: expand_channel_meta from 'info' to 'info' 2024-05-04T12:10:39.542 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: expand_channel_meta expanded map: {default=info} 2024-05-04T12:10:39.542 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: expand_channel_meta expand map: {default=daemon} 2024-05-04T12:10:39.542 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: expand_channel_meta from 'daemon' to 'daemon' 2024-05-04T12:10:39.543 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: expand_channel_meta expanded map: {default=daemon} 2024-05-04T12:10:39.543 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: expand_channel_meta expand map: {default=debug} 2024-05-04T12:10:39.543 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: expand_channel_meta from 'debug' to 'debug' 2024-05-04T12:10:39.543 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 ceph-mon[35311]: expand_channel_meta expanded map: {default=debug} 2024-05-04T12:10:39.543 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 bash[35264]: 085b6b3257eecaf5105784eeb8c64a3be9070d608f6dffc215de00b5b5d2e2a4 2024-05-04T12:10:39.543 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:39 smithi146 systemd[1]: Started Ceph mon.smithi146 for d9b985c4-0a0e-11ef-bc97-c7b262605968. 2024-05-04T12:10:39.559 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:39.559+0000 7f67f5190700 1 -- 172.21.15.146:0/1968353948 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f67f0068f90 con 0x7f67f0105700 2024-05-04T12:10:44.512 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-05-04T12:10:44.512 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":2,"fsid":"d9b985c4-0a0e-11ef-bc97-c7b262605968","modified":"2024-05-04T12:10:39.495592Z","created":"2024-05-04T12:07:48.391828Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi079","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:3300","nonce":0},{"type":"v1","addr":"172.21.15.79:6789","nonce":0}]},"addr":"172.21.15.79:6789/0","public_addr":"172.21.15.79:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"smithi146","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:3300","nonce":0},{"type":"v1","addr":"172.21.15.146:6789","nonce":0}]},"addr":"172.21.15.146:6789/0","public_addr":"172.21.15.146:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-05-04T12:10:44.512 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:44.512+0000 7f67d7fff700 1 -- 172.21.15.146:0/1968353948 <== mon.0 v2:172.21.15.79:3300/0 8 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+1022 (secure 0 0 0) 0x7f67e002a360 con 0x7f67f0105700 2024-05-04T12:10:44.514 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:44.516+0000 7f67f5190700 1 -- 172.21.15.146:0/1968353948 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f67d80384b0 msgr2=0x7f67d803a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:10:44.514 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:44.516+0000 7f67f5190700 1 --2- 172.21.15.146:0/1968353948 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f67d80384b0 0x7f67d803a970 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f67e4006fd0 tx=0x7f67e4006e40 comp rx=0 tx=0).stop 2024-05-04T12:10:44.514 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:44.516+0000 7f67f5190700 1 -- 172.21.15.146:0/1968353948 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f67f0105700 msgr2=0x7f67f0196b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:10:44.514 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:44.516+0000 7f67f5190700 1 --2- 172.21.15.146:0/1968353948 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f67f0105700 0x7f67f0196b40 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f67e0005b40 tx=0x7f67e0004c80 comp rx=0 tx=0).stop 2024-05-04T12:10:44.514 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:44.516+0000 7f67f5190700 1 -- 172.21.15.146:0/1968353948 shutdown_connections 2024-05-04T12:10:44.514 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:44.516+0000 7f67f5190700 1 --2- 172.21.15.146:0/1968353948 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f67d80384b0 0x7f67d803a970 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:10:44.514 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:44.516+0000 7f67f5190700 1 --2- 172.21.15.146:0/1968353948 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f67f0105700 0x7f67f0196b40 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:10:44.515 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:44.516+0000 7f67f5190700 1 -- 172.21.15.146:0/1968353948 >> 172.21.15.146:0/1968353948 conn(0x7f67f00fb210 msgr2=0x7f67f0107b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:10:44.515 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:44.516+0000 7f67f5190700 1 -- 172.21.15.146:0/1968353948 shutdown_connections 2024-05-04T12:10:44.515 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:10:44.516+0000 7f67f5190700 1 -- 172.21.15.146:0/1968353948 wait complete. 2024-05-04T12:10:44.515 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 2 2024-05-04T12:10:44.777 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:44 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "mon metadata", "id": "smithi079"}]: dispatch 2024-05-04T12:10:44.777 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:44 smithi146 ceph-mon[35311]: mon.smithi079 calling monitor election 2024-05-04T12:10:44.777 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:44 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "mon metadata", "id": "smithi146"}]: dispatch 2024-05-04T12:10:44.777 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:44 smithi146 ceph-mon[35311]: from='client.? 172.21.15.146:0/1968353948' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-04T12:10:44.777 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:44 smithi146 ceph-mon[35311]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-04T12:10:44.777 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:44 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "mon metadata", "id": "smithi146"}]: dispatch 2024-05-04T12:10:44.777 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:44 smithi146 ceph-mon[35311]: from='mgr.? 172.21.15.146:0/468595484' entity='mgr.smithi146.eubqih' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi146.eubqih/crt"}]: dispatch 2024-05-04T12:10:44.778 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:44 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "mon metadata", "id": "smithi146"}]: dispatch 2024-05-04T12:10:44.778 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:44 smithi146 ceph-mon[35311]: mon.smithi146 calling monitor election 2024-05-04T12:10:44.778 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:44 smithi146 ceph-mon[35311]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-04T12:10:44.778 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:44 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-04T12:10:44.778 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:44 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "mon metadata", "id": "smithi146"}]: dispatch 2024-05-04T12:10:44.778 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:44 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "mon metadata", "id": "smithi146"}]: dispatch 2024-05-04T12:10:44.778 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:44 smithi146 ceph-mon[35311]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-04T12:10:44.778 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:44 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "mon metadata", "id": "smithi146"}]: dispatch 2024-05-04T12:10:44.778 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:44 smithi146 ceph-mon[35311]: mon.smithi079 is new leader, mons smithi079,smithi146 in quorum (ranks 0,1) 2024-05-04T12:10:44.778 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:44 smithi146 ceph-mon[35311]: monmap e2: 2 mons at {smithi079=[v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0],smithi146=[v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-05-04T12:10:44.778 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:44 smithi146 ceph-mon[35311]: fsmap 2024-05-04T12:10:44.778 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:44 smithi146 ceph-mon[35311]: osdmap e5: 0 total, 0 up, 0 in 2024-05-04T12:10:44.778 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:44 smithi146 ceph-mon[35311]: mgrmap e17: smithi079.dgcpwl(active, since 32s) 2024-05-04T12:10:44.778 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:44 smithi146 ceph-mon[35311]: Standby manager daemon smithi146.eubqih started 2024-05-04T12:10:44.779 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:44 smithi146 ceph-mon[35311]: from='mgr.? 172.21.15.146:0/468595484' entity='mgr.smithi146.eubqih' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-04T12:10:44.779 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:44 smithi146 ceph-mon[35311]: overall HEALTH_OK 2024-05-04T12:10:44.779 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:44 smithi146 ceph-mon[35311]: from='mgr.? 172.21.15.146:0/468595484' entity='mgr.smithi146.eubqih' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi146.eubqih/key"}]: dispatch 2024-05-04T12:10:44.779 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:44 smithi146 ceph-mon[35311]: from='mgr.? 172.21.15.146:0/468595484' entity='mgr.smithi146.eubqih' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-04T12:10:44.779 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:44 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:10:44.779 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:44 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:10:44.779 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:44 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:10:44.779 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:44 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:10:44.779 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:44 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:10:44.945 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:44 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "mon metadata", "id": "smithi079"}]: dispatch 2024-05-04T12:10:44.945 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:44 smithi079 ceph-mon[27222]: mon.smithi079 calling monitor election 2024-05-04T12:10:44.945 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:44 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "mon metadata", "id": "smithi146"}]: dispatch 2024-05-04T12:10:44.945 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:44 smithi079 ceph-mon[27222]: from='client.? 172.21.15.146:0/1968353948' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-04T12:10:44.946 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:44 smithi079 ceph-mon[27222]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-04T12:10:44.946 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:44 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "mon metadata", "id": "smithi146"}]: dispatch 2024-05-04T12:10:44.946 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:44 smithi079 ceph-mon[27222]: from='mgr.? 172.21.15.146:0/468595484' entity='mgr.smithi146.eubqih' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi146.eubqih/crt"}]: dispatch 2024-05-04T12:10:44.946 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:44 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "mon metadata", "id": "smithi146"}]: dispatch 2024-05-04T12:10:44.946 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:44 smithi079 ceph-mon[27222]: mon.smithi146 calling monitor election 2024-05-04T12:10:44.946 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:44 smithi079 ceph-mon[27222]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-04T12:10:44.946 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:44 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-04T12:10:44.946 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:44 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "mon metadata", "id": "smithi146"}]: dispatch 2024-05-04T12:10:44.946 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:44 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "mon metadata", "id": "smithi146"}]: dispatch 2024-05-04T12:10:44.946 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:44 smithi079 ceph-mon[27222]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-04T12:10:44.946 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:44 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "mon metadata", "id": "smithi146"}]: dispatch 2024-05-04T12:10:44.946 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:44 smithi079 ceph-mon[27222]: mon.smithi079 is new leader, mons smithi079,smithi146 in quorum (ranks 0,1) 2024-05-04T12:10:44.946 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:44 smithi079 ceph-mon[27222]: monmap e2: 2 mons at {smithi079=[v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0],smithi146=[v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-05-04T12:10:44.947 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:44 smithi079 ceph-mon[27222]: fsmap 2024-05-04T12:10:44.947 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:44 smithi079 ceph-mon[27222]: osdmap e5: 0 total, 0 up, 0 in 2024-05-04T12:10:44.947 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:44 smithi079 ceph-mon[27222]: mgrmap e17: smithi079.dgcpwl(active, since 32s) 2024-05-04T12:10:44.947 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:44 smithi079 ceph-mon[27222]: Standby manager daemon smithi146.eubqih started 2024-05-04T12:10:44.947 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:44 smithi079 ceph-mon[27222]: from='mgr.? 172.21.15.146:0/468595484' entity='mgr.smithi146.eubqih' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-04T12:10:44.947 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:44 smithi079 ceph-mon[27222]: overall HEALTH_OK 2024-05-04T12:10:44.947 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:44 smithi079 ceph-mon[27222]: from='mgr.? 172.21.15.146:0/468595484' entity='mgr.smithi146.eubqih' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi146.eubqih/key"}]: dispatch 2024-05-04T12:10:44.947 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:44 smithi079 ceph-mon[27222]: from='mgr.? 172.21.15.146:0/468595484' entity='mgr.smithi146.eubqih' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-04T12:10:44.947 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:44 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:10:44.947 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:44 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:10:44.947 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:44 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:10:44.947 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:44 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:10:44.947 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:44 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:10:45.161 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-05-04T12:10:45.161 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d9b985c4-0a0e-11ef-bc97-c7b262605968 -- ceph config generate-minimal-conf 2024-05-04T12:10:45.380 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/d9b985c4-0a0e-11ef-bc97-c7b262605968/mon.smithi079/config 2024-05-04T12:10:45.777 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:45 smithi146 ceph-mon[35311]: mgrmap e18: smithi079.dgcpwl(active, since 32s), standbys: smithi146.eubqih 2024-05-04T12:10:45.777 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:45 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "mgr metadata", "who": "smithi146.eubqih", "id": "smithi146.eubqih"}]: dispatch 2024-05-04T12:10:45.777 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:45 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-04T12:10:45.777 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:45 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "mon metadata", "id": "smithi146"}]: dispatch 2024-05-04T12:10:45.843 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:45 smithi079 ceph-mon[27222]: mgrmap e18: smithi079.dgcpwl(active, since 32s), standbys: smithi146.eubqih 2024-05-04T12:10:45.843 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:45 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "mgr metadata", "who": "smithi146.eubqih", "id": "smithi146.eubqih"}]: dispatch 2024-05-04T12:10:45.843 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:45 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-04T12:10:45.843 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:45 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "mon metadata", "id": "smithi146"}]: dispatch 2024-05-04T12:10:46.949 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:10:46.949+0000 7f485a383700 1 -- 172.21.15.79:0/1937611024 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4854100090 msgr2=0x7f4854100470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:10:46.949 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:10:46.949+0000 7f485a383700 1 --2- 172.21.15.79:0/1937611024 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4854100090 0x7f4854100470 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f4844009910 tx=0x7f4844009c20 comp rx=0 tx=0).stop 2024-05-04T12:10:46.949 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:10:46.949+0000 7f485a383700 1 -- 172.21.15.79:0/1937611024 shutdown_connections 2024-05-04T12:10:46.949 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:10:46.949+0000 7f485a383700 1 --2- 172.21.15.79:0/1937611024 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4854100090 0x7f4854100470 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:10:46.949 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:10:46.949+0000 7f485a383700 1 -- 172.21.15.79:0/1937611024 >> 172.21.15.79:0/1937611024 conn(0x7f4854075df0 msgr2=0x7f4854076200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:10:46.949 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:10:46.950+0000 7f485a383700 1 -- 172.21.15.79:0/1937611024 shutdown_connections 2024-05-04T12:10:46.949 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:10:46.950+0000 7f485a383700 1 -- 172.21.15.79:0/1937611024 wait complete. 2024-05-04T12:10:46.949 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:10:46.950+0000 7f485a383700 1 Processor -- start 2024-05-04T12:10:46.950 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:10:46.950+0000 7f485a383700 1 -- start start 2024-05-04T12:10:46.951 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:10:46.951+0000 7f485a383700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4854100090 0x7f4854194d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:10:46.951 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:10:46.951+0000 7f485a383700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f48541952c0 0x7f4854199730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:10:46.951 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:10:46.951+0000 7f485a383700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48541958e0 con 0x7f4854100090 2024-05-04T12:10:46.951 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:10:46.951+0000 7f485a383700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4854195a50 con 0x7f48541952c0 2024-05-04T12:10:46.951 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:10:46.951+0000 7f4853fff700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4854100090 0x7f4854194d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:10:46.951 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:10:46.951+0000 7f4853fff700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4854100090 0x7f4854194d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:57688/0 (socket says 172.21.15.79:57688) 2024-05-04T12:10:46.951 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:10:46.951+0000 7f4853fff700 1 -- 172.21.15.79:0/3220567811 learned_addr learned my addr 172.21.15.79:0/3220567811 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:10:46.952 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:10:46.951+0000 7f48537fe700 1 --2- 172.21.15.79:0/3220567811 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f48541952c0 0x7f4854199730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:10:46.952 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:10:46.951+0000 7f4853fff700 1 -- 172.21.15.79:0/3220567811 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f48541952c0 msgr2=0x7f4854199730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:10:46.952 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:10:46.951+0000 7f4853fff700 1 --2- 172.21.15.79:0/3220567811 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f48541952c0 0x7f4854199730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:10:46.952 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:10:46.951+0000 7f4853fff700 1 -- 172.21.15.79:0/3220567811 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48440095f0 con 0x7f4854100090 2024-05-04T12:10:46.952 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:10:46.951+0000 7f4853fff700 1 --2- 172.21.15.79:0/3220567811 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4854100090 0x7f4854194d80 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f4844006010 tx=0x7f484400bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:10:46.952 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:10:46.951+0000 7f48517fa700 1 -- 172.21.15.79:0/3220567811 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f484401f070 con 0x7f4854100090 2024-05-04T12:10:46.952 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:10:46.952+0000 7f48517fa700 1 -- 172.21.15.79:0/3220567811 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f4844005140 con 0x7f4854100090 2024-05-04T12:10:46.952 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:10:46.952+0000 7f485a383700 1 -- 172.21.15.79:0/3220567811 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4854199cd0 con 0x7f4854100090 2024-05-04T12:10:46.952 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:10:46.952+0000 7f48517fa700 1 -- 172.21.15.79:0/3220567811 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f484400f690 con 0x7f4854100090 2024-05-04T12:10:46.952 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:10:46.952+0000 7f485a383700 1 -- 172.21.15.79:0/3220567811 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f485419a2a0 con 0x7f4854100090 2024-05-04T12:10:46.952 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:10:46.952+0000 7f485a383700 1 -- 172.21.15.79:0/3220567811 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48540623c0 con 0x7f4854100090 2024-05-04T12:10:46.953 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:10:46.953+0000 7f48517fa700 1 -- 172.21.15.79:0/3220567811 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 18) v1 ==== 90256+0+0 (secure 0 0 0) 0x7f4844019650 con 0x7f4854100090 2024-05-04T12:10:46.953 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:10:46.954+0000 7f48517fa700 1 --2- 172.21.15.79:0/3220567811 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f483c06c5c0 0x7f483c06ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:10:46.953 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:10:46.954+0000 7f48517fa700 1 -- 172.21.15.79:0/3220567811 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7f484408df70 con 0x7f4854100090 2024-05-04T12:10:46.953 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:10:46.954+0000 7f48537fe700 1 --2- 172.21.15.79:0/3220567811 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f483c06c5c0 0x7f483c06ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:10:46.953 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:10:46.954+0000 7f48537fe700 1 --2- 172.21.15.79:0/3220567811 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f483c06c5c0 0x7f483c06ea80 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f4848009850 tx=0x7f4848008040 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:10:46.957 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:10:46.958+0000 7f48517fa700 1 -- 172.21.15.79:0/3220567811 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f4844059950 con 0x7f4854100090 2024-05-04T12:10:47.156 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:10:47.156+0000 7f485a383700 1 -- 172.21.15.79:0/3220567811 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f4854103b00 con 0x7f4854100090 2024-05-04T12:10:47.156 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:10:47.157+0000 7f48517fa700 1 -- 172.21.15.79:0/3220567811 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v10) v1 ==== 76+0+225 (secure 0 0 0) 0x7f4844028020 con 0x7f4854100090 2024-05-04T12:10:47.156 INFO:teuthology.orchestra.run.smithi079.stdout:# minimal ceph.conf for d9b985c4-0a0e-11ef-bc97-c7b262605968 2024-05-04T12:10:47.156 INFO:teuthology.orchestra.run.smithi079.stdout:[global] 2024-05-04T12:10:47.157 INFO:teuthology.orchestra.run.smithi079.stdout: fsid = d9b985c4-0a0e-11ef-bc97-c7b262605968 2024-05-04T12:10:47.157 INFO:teuthology.orchestra.run.smithi079.stdout: mon_host = [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] 2024-05-04T12:10:47.158 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:10:47.159+0000 7f485a383700 1 -- 172.21.15.79:0/3220567811 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f483c06c5c0 msgr2=0x7f483c06ea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:10:47.158 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:10:47.159+0000 7f485a383700 1 --2- 172.21.15.79:0/3220567811 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f483c06c5c0 0x7f483c06ea80 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f4848009850 tx=0x7f4848008040 comp rx=0 tx=0).stop 2024-05-04T12:10:47.158 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:10:47.159+0000 7f485a383700 1 -- 172.21.15.79:0/3220567811 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4854100090 msgr2=0x7f4854194d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:10:47.158 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:10:47.159+0000 7f485a383700 1 --2- 172.21.15.79:0/3220567811 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4854100090 0x7f4854194d80 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f4844006010 tx=0x7f484400bda0 comp rx=0 tx=0).stop 2024-05-04T12:10:47.158 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:10:47.159+0000 7f485a383700 1 -- 172.21.15.79:0/3220567811 shutdown_connections 2024-05-04T12:10:47.158 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:10:47.159+0000 7f485a383700 1 --2- 172.21.15.79:0/3220567811 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f483c06c5c0 0x7f483c06ea80 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:10:47.158 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:10:47.159+0000 7f485a383700 1 --2- 172.21.15.79:0/3220567811 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4854100090 0x7f4854194d80 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:10:47.158 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:10:47.159+0000 7f485a383700 1 --2- 172.21.15.79:0/3220567811 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f48541952c0 0x7f4854199730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:10:47.158 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:10:47.159+0000 7f485a383700 1 -- 172.21.15.79:0/3220567811 >> 172.21.15.79:0/3220567811 conn(0x7f4854075df0 msgr2=0x7f4854101260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:10:47.159 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:10:47.159+0000 7f485a383700 1 -- 172.21.15.79:0/3220567811 shutdown_connections 2024-05-04T12:10:47.159 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:10:47.159+0000 7f485a383700 1 -- 172.21.15.79:0/3220567811 wait complete. 2024-05-04T12:10:47.276 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:47 smithi146 ceph-mon[35311]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-04T12:10:47.276 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:47 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:10:47.276 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:47 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:10:47.277 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:47 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:10:47.277 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:47 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-04T12:10:47.277 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:47 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-04T12:10:47.281 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:46 smithi079 ceph-mon[27222]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-04T12:10:47.281 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:46 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:10:47.282 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:46 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:10:47.282 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:46 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:10:47.282 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:46 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-04T12:10:47.282 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:46 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-04T12:10:47.811 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-05-04T12:10:47.811 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-05-04T12:10:47.811 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/etc/ceph/ceph.conf 2024-05-04T12:10:47.839 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-05-04T12:10:47.839 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-05-04T12:10:47.902 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-05-04T12:10:47.902 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/etc/ceph/ceph.conf 2024-05-04T12:10:47.930 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-05-04T12:10:47.930 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-05-04T12:10:47.995 INFO:tasks.cephadm:Deploying OSDs... 2024-05-04T12:10:47.995 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-05-04T12:10:47.995 DEBUG:teuthology.orchestra.run.smithi079:> dd if=/scratch_devs of=/dev/stdout 2024-05-04T12:10:48.010 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-05-04T12:10:48.010 DEBUG:teuthology.orchestra.run.smithi079:> stat /dev/vg_nvme/lv_1 2024-05-04T12:10:48.049 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:47 smithi079 ceph-mon[27222]: Updating smithi079:/etc/ceph/ceph.conf 2024-05-04T12:10:48.049 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:47 smithi079 ceph-mon[27222]: Updating smithi146:/etc/ceph/ceph.conf 2024-05-04T12:10:48.049 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:47 smithi079 ceph-mon[27222]: Updating smithi079:/var/lib/ceph/d9b985c4-0a0e-11ef-bc97-c7b262605968/config/ceph.conf 2024-05-04T12:10:48.049 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:47 smithi079 ceph-mon[27222]: Updating smithi146:/var/lib/ceph/d9b985c4-0a0e-11ef-bc97-c7b262605968/config/ceph.conf 2024-05-04T12:10:48.049 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:47 smithi079 ceph-mon[27222]: from='client.? 172.21.15.79:0/3220567811' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-04T12:10:48.050 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:47 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:10:48.050 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:47 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:10:48.050 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:47 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:10:48.050 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:47 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:10:48.050 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:47 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:10:48.050 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:47 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-04T12:10:48.051 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:47 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-04T12:10:48.051 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:47 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-04T12:10:48.066 INFO:teuthology.orchestra.run.smithi079.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-05-04T12:10:48.066 INFO:teuthology.orchestra.run.smithi079.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-04T12:10:48.066 INFO:teuthology.orchestra.run.smithi079.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-05-04T12:10:48.066 INFO:teuthology.orchestra.run.smithi079.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-04T12:10:48.066 INFO:teuthology.orchestra.run.smithi079.stdout:Context: system_u:object_r:device_t:s0 2024-05-04T12:10:48.066 INFO:teuthology.orchestra.run.smithi079.stdout:Access: 2024-05-04 12:10:03.368427091 +0000 2024-05-04T12:10:48.067 INFO:teuthology.orchestra.run.smithi079.stdout:Modify: 2024-05-04 12:03:57.314124573 +0000 2024-05-04T12:10:48.067 INFO:teuthology.orchestra.run.smithi079.stdout:Change: 2024-05-04 12:03:57.314124573 +0000 2024-05-04T12:10:48.067 INFO:teuthology.orchestra.run.smithi079.stdout: Birth: 2024-05-04 12:03:57.314124573 +0000 2024-05-04T12:10:48.067 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-05-04T12:10:48.130 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records in 2024-05-04T12:10:48.130 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records out 2024-05-04T12:10:48.131 INFO:teuthology.orchestra.run.smithi079.stderr:512 bytes copied, 0.000267005 s, 1.9 MB/s 2024-05-04T12:10:48.132 DEBUG:teuthology.orchestra.run.smithi079:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-05-04T12:10:48.189 DEBUG:teuthology.orchestra.run.smithi079:> stat /dev/vg_nvme/lv_2 2024-05-04T12:10:48.244 INFO:teuthology.orchestra.run.smithi079.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-05-04T12:10:48.244 INFO:teuthology.orchestra.run.smithi079.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-04T12:10:48.244 INFO:teuthology.orchestra.run.smithi079.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-05-04T12:10:48.244 INFO:teuthology.orchestra.run.smithi079.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-04T12:10:48.244 INFO:teuthology.orchestra.run.smithi079.stdout:Context: system_u:object_r:device_t:s0 2024-05-04T12:10:48.244 INFO:teuthology.orchestra.run.smithi079.stdout:Access: 2024-05-04 12:10:03.369427069 +0000 2024-05-04T12:10:48.244 INFO:teuthology.orchestra.run.smithi079.stdout:Modify: 2024-05-04 12:03:57.657116377 +0000 2024-05-04T12:10:48.244 INFO:teuthology.orchestra.run.smithi079.stdout:Change: 2024-05-04 12:03:57.657116377 +0000 2024-05-04T12:10:48.244 INFO:teuthology.orchestra.run.smithi079.stdout: Birth: 2024-05-04 12:03:57.657116377 +0000 2024-05-04T12:10:48.245 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-05-04T12:10:48.277 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:47 smithi146 ceph-mon[35311]: Updating smithi079:/etc/ceph/ceph.conf 2024-05-04T12:10:48.277 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:47 smithi146 ceph-mon[35311]: Updating smithi146:/etc/ceph/ceph.conf 2024-05-04T12:10:48.277 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:47 smithi146 ceph-mon[35311]: Updating smithi079:/var/lib/ceph/d9b985c4-0a0e-11ef-bc97-c7b262605968/config/ceph.conf 2024-05-04T12:10:48.277 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:47 smithi146 ceph-mon[35311]: Updating smithi146:/var/lib/ceph/d9b985c4-0a0e-11ef-bc97-c7b262605968/config/ceph.conf 2024-05-04T12:10:48.277 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:47 smithi146 ceph-mon[35311]: from='client.? 172.21.15.79:0/3220567811' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-04T12:10:48.277 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:47 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:10:48.277 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:47 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:10:48.277 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:47 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:10:48.277 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:47 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:10:48.277 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:47 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:10:48.277 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:47 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-04T12:10:48.277 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:47 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-04T12:10:48.278 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:48 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-04T12:10:48.307 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records in 2024-05-04T12:10:48.307 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records out 2024-05-04T12:10:48.307 INFO:teuthology.orchestra.run.smithi079.stderr:512 bytes copied, 0.000314726 s, 1.6 MB/s 2024-05-04T12:10:48.309 DEBUG:teuthology.orchestra.run.smithi079:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-05-04T12:10:48.366 DEBUG:teuthology.orchestra.run.smithi079:> stat /dev/vg_nvme/lv_3 2024-05-04T12:10:48.420 INFO:teuthology.orchestra.run.smithi079.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-05-04T12:10:48.420 INFO:teuthology.orchestra.run.smithi079.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-04T12:10:48.420 INFO:teuthology.orchestra.run.smithi079.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-05-04T12:10:48.420 INFO:teuthology.orchestra.run.smithi079.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-04T12:10:48.420 INFO:teuthology.orchestra.run.smithi079.stdout:Context: system_u:object_r:device_t:s0 2024-05-04T12:10:48.420 INFO:teuthology.orchestra.run.smithi079.stdout:Access: 2024-05-04 12:10:03.369427069 +0000 2024-05-04T12:10:48.420 INFO:teuthology.orchestra.run.smithi079.stdout:Modify: 2024-05-04 12:03:58.009107965 +0000 2024-05-04T12:10:48.420 INFO:teuthology.orchestra.run.smithi079.stdout:Change: 2024-05-04 12:03:58.009107965 +0000 2024-05-04T12:10:48.421 INFO:teuthology.orchestra.run.smithi079.stdout: Birth: 2024-05-04 12:03:58.009107965 +0000 2024-05-04T12:10:48.421 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-05-04T12:10:48.482 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records in 2024-05-04T12:10:48.482 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records out 2024-05-04T12:10:48.482 INFO:teuthology.orchestra.run.smithi079.stderr:512 bytes copied, 0.000182421 s, 2.8 MB/s 2024-05-04T12:10:48.484 DEBUG:teuthology.orchestra.run.smithi079:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-05-04T12:10:48.539 DEBUG:teuthology.orchestra.run.smithi079:> stat /dev/vg_nvme/lv_4 2024-05-04T12:10:48.594 INFO:teuthology.orchestra.run.smithi079.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-05-04T12:10:48.594 INFO:teuthology.orchestra.run.smithi079.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-04T12:10:48.595 INFO:teuthology.orchestra.run.smithi079.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-05-04T12:10:48.595 INFO:teuthology.orchestra.run.smithi079.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-04T12:10:48.595 INFO:teuthology.orchestra.run.smithi079.stdout:Context: system_u:object_r:device_t:s0 2024-05-04T12:10:48.595 INFO:teuthology.orchestra.run.smithi079.stdout:Access: 2024-05-04 12:10:03.369427069 +0000 2024-05-04T12:10:48.595 INFO:teuthology.orchestra.run.smithi079.stdout:Modify: 2024-05-04 12:03:58.346099912 +0000 2024-05-04T12:10:48.595 INFO:teuthology.orchestra.run.smithi079.stdout:Change: 2024-05-04 12:03:58.346099912 +0000 2024-05-04T12:10:48.595 INFO:teuthology.orchestra.run.smithi079.stdout: Birth: 2024-05-04 12:03:58.346099912 +0000 2024-05-04T12:10:48.595 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-05-04T12:10:48.657 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records in 2024-05-04T12:10:48.657 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records out 2024-05-04T12:10:48.657 INFO:teuthology.orchestra.run.smithi079.stderr:512 bytes copied, 0.000177282 s, 2.9 MB/s 2024-05-04T12:10:48.658 DEBUG:teuthology.orchestra.run.smithi079:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-05-04T12:10:48.715 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-05-04T12:10:48.715 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/scratch_devs of=/dev/stdout 2024-05-04T12:10:48.734 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-05-04T12:10:48.734 DEBUG:teuthology.orchestra.run.smithi146:> stat /dev/vg_nvme/lv_1 2024-05-04T12:10:48.792 INFO:teuthology.orchestra.run.smithi146.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-05-04T12:10:48.792 INFO:teuthology.orchestra.run.smithi146.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-04T12:10:48.792 INFO:teuthology.orchestra.run.smithi146.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-05-04T12:10:48.792 INFO:teuthology.orchestra.run.smithi146.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-04T12:10:48.792 INFO:teuthology.orchestra.run.smithi146.stdout:Context: system_u:object_r:device_t:s0 2024-05-04T12:10:48.792 INFO:teuthology.orchestra.run.smithi146.stdout:Access: 2024-05-04 12:10:38.162516697 +0000 2024-05-04T12:10:48.792 INFO:teuthology.orchestra.run.smithi146.stdout:Modify: 2024-05-04 12:04:08.022457387 +0000 2024-05-04T12:10:48.792 INFO:teuthology.orchestra.run.smithi146.stdout:Change: 2024-05-04 12:04:08.022457387 +0000 2024-05-04T12:10:48.792 INFO:teuthology.orchestra.run.smithi146.stdout: Birth: 2024-05-04 12:04:08.022457387 +0000 2024-05-04T12:10:48.792 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-05-04T12:10:48.858 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records in 2024-05-04T12:10:48.858 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records out 2024-05-04T12:10:48.858 INFO:teuthology.orchestra.run.smithi146.stderr:512 bytes copied, 0.000235709 s, 2.2 MB/s 2024-05-04T12:10:48.860 DEBUG:teuthology.orchestra.run.smithi146:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-05-04T12:10:48.918 DEBUG:teuthology.orchestra.run.smithi146:> stat /dev/vg_nvme/lv_2 2024-05-04T12:10:48.974 INFO:teuthology.orchestra.run.smithi146.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-05-04T12:10:48.974 INFO:teuthology.orchestra.run.smithi146.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-04T12:10:48.974 INFO:teuthology.orchestra.run.smithi146.stdout:Device: 5h/5d Inode: 683 Links: 1 2024-05-04T12:10:48.974 INFO:teuthology.orchestra.run.smithi146.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-04T12:10:48.974 INFO:teuthology.orchestra.run.smithi146.stdout:Context: system_u:object_r:device_t:s0 2024-05-04T12:10:48.975 INFO:teuthology.orchestra.run.smithi146.stdout:Access: 2024-05-04 12:10:38.162516697 +0000 2024-05-04T12:10:48.975 INFO:teuthology.orchestra.run.smithi146.stdout:Modify: 2024-05-04 12:04:08.356451826 +0000 2024-05-04T12:10:48.975 INFO:teuthology.orchestra.run.smithi146.stdout:Change: 2024-05-04 12:04:08.356451826 +0000 2024-05-04T12:10:48.975 INFO:teuthology.orchestra.run.smithi146.stdout: Birth: 2024-05-04 12:04:08.356451826 +0000 2024-05-04T12:10:48.975 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-05-04T12:10:49.039 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records in 2024-05-04T12:10:49.039 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records out 2024-05-04T12:10:49.039 INFO:teuthology.orchestra.run.smithi146.stderr:512 bytes copied, 0.000211183 s, 2.4 MB/s 2024-05-04T12:10:49.041 DEBUG:teuthology.orchestra.run.smithi146:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-05-04T12:10:49.098 DEBUG:teuthology.orchestra.run.smithi146:> stat /dev/vg_nvme/lv_3 2024-05-04T12:10:49.153 INFO:teuthology.orchestra.run.smithi146.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-05-04T12:10:49.154 INFO:teuthology.orchestra.run.smithi146.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-04T12:10:49.154 INFO:teuthology.orchestra.run.smithi146.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-05-04T12:10:49.154 INFO:teuthology.orchestra.run.smithi146.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-04T12:10:49.154 INFO:teuthology.orchestra.run.smithi146.stdout:Context: system_u:object_r:device_t:s0 2024-05-04T12:10:49.154 INFO:teuthology.orchestra.run.smithi146.stdout:Access: 2024-05-04 12:10:38.162516697 +0000 2024-05-04T12:10:49.154 INFO:teuthology.orchestra.run.smithi146.stdout:Modify: 2024-05-04 12:04:08.696446166 +0000 2024-05-04T12:10:49.154 INFO:teuthology.orchestra.run.smithi146.stdout:Change: 2024-05-04 12:04:08.696446166 +0000 2024-05-04T12:10:49.154 INFO:teuthology.orchestra.run.smithi146.stdout: Birth: 2024-05-04 12:04:08.696446166 +0000 2024-05-04T12:10:49.155 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-05-04T12:10:49.195 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:49 smithi079 ceph-mon[27222]: Reconfiguring mon.smithi079 (unknown last config time)... 2024-05-04T12:10:49.195 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:49 smithi079 ceph-mon[27222]: Reconfiguring daemon mon.smithi079 on smithi079 2024-05-04T12:10:49.195 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:49 smithi079 ceph-mon[27222]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-04T12:10:49.195 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:49 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:10:49.195 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:49 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:10:49.216 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:49 smithi146 ceph-mon[35311]: Reconfiguring mon.smithi079 (unknown last config time)... 2024-05-04T12:10:49.216 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:49 smithi146 ceph-mon[35311]: Reconfiguring daemon mon.smithi079 on smithi079 2024-05-04T12:10:49.216 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:49 smithi146 ceph-mon[35311]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-04T12:10:49.216 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:49 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:10:49.216 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:49 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:10:49.218 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records in 2024-05-04T12:10:49.218 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records out 2024-05-04T12:10:49.218 INFO:teuthology.orchestra.run.smithi146.stderr:512 bytes copied, 0.00026105 s, 2.0 MB/s 2024-05-04T12:10:49.219 DEBUG:teuthology.orchestra.run.smithi146:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-05-04T12:10:49.276 DEBUG:teuthology.orchestra.run.smithi146:> stat /dev/vg_nvme/lv_4 2024-05-04T12:10:49.332 INFO:teuthology.orchestra.run.smithi146.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-05-04T12:10:49.332 INFO:teuthology.orchestra.run.smithi146.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-04T12:10:49.332 INFO:teuthology.orchestra.run.smithi146.stdout:Device: 5h/5d Inode: 721 Links: 1 2024-05-04T12:10:49.332 INFO:teuthology.orchestra.run.smithi146.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-04T12:10:49.332 INFO:teuthology.orchestra.run.smithi146.stdout:Context: system_u:object_r:device_t:s0 2024-05-04T12:10:49.332 INFO:teuthology.orchestra.run.smithi146.stdout:Access: 2024-05-04 12:10:38.163516682 +0000 2024-05-04T12:10:49.332 INFO:teuthology.orchestra.run.smithi146.stdout:Modify: 2024-05-04 12:04:09.028440638 +0000 2024-05-04T12:10:49.332 INFO:teuthology.orchestra.run.smithi146.stdout:Change: 2024-05-04 12:04:09.028440638 +0000 2024-05-04T12:10:49.333 INFO:teuthology.orchestra.run.smithi146.stdout: Birth: 2024-05-04 12:04:09.028440638 +0000 2024-05-04T12:10:49.333 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-05-04T12:10:49.394 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records in 2024-05-04T12:10:49.394 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records out 2024-05-04T12:10:49.394 INFO:teuthology.orchestra.run.smithi146.stderr:512 bytes copied, 0.000178207 s, 2.9 MB/s 2024-05-04T12:10:49.395 DEBUG:teuthology.orchestra.run.smithi146:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-05-04T12:10:49.451 INFO:tasks.cephadm:Deploying osd.0 on smithi079 with /dev/vg_nvme/lv_4... 2024-05-04T12:10:49.451 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d9b985c4-0a0e-11ef-bc97-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-05-04T12:10:50.276 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:50 smithi146 ceph-mon[35311]: Reconfiguring mgr.smithi079.dgcpwl (unknown last config time)... 2024-05-04T12:10:50.276 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:50 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi079.dgcpwl", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-04T12:10:50.277 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:50 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-04T12:10:50.277 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:50 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-04T12:10:50.277 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:50 smithi146 ceph-mon[35311]: Reconfiguring daemon mgr.smithi079.dgcpwl on smithi079 2024-05-04T12:10:50.289 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:50 smithi079 ceph-mon[27222]: Reconfiguring mgr.smithi079.dgcpwl (unknown last config time)... 2024-05-04T12:10:50.289 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:50 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi079.dgcpwl", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-04T12:10:50.289 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:50 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-04T12:10:50.289 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:50 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-04T12:10:50.289 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:50 smithi079 ceph-mon[27222]: Reconfiguring daemon mgr.smithi079.dgcpwl on smithi079 2024-05-04T12:10:50.387 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/d9b985c4-0a0e-11ef-bc97-c7b262605968/mon.smithi079/config 2024-05-04T12:10:51.272 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:51 smithi146 ceph-mon[35311]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-04T12:10:51.272 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:51 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:10:51.272 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:51 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:10:51.272 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:51 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi079", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-05-04T12:10:51.272 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:51 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-04T12:10:51.300 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:51 smithi079 ceph-mon[27222]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-04T12:10:51.301 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:51 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:10:51.301 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:51 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:10:51.301 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:51 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi079", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-05-04T12:10:51.301 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:51 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-04T12:10:52.276 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:52 smithi146 ceph-mon[35311]: Reconfiguring ceph-exporter.smithi079 (monmap changed)... 2024-05-04T12:10:52.277 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:52 smithi146 ceph-mon[35311]: Reconfiguring daemon ceph-exporter.smithi079 on smithi079 2024-05-04T12:10:52.293 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:52 smithi079 ceph-mon[27222]: Reconfiguring ceph-exporter.smithi079 (monmap changed)... 2024-05-04T12:10:52.293 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:52 smithi079 ceph-mon[27222]: Reconfiguring daemon ceph-exporter.smithi079 on smithi079 2024-05-04T12:10:53.104 INFO:teuthology.orchestra.run.smithi079.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-05-04T12:10:53.104 INFO:teuthology.orchestra.run.smithi079.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-05-04T12:10:53.104 INFO:teuthology.orchestra.run.smithi079.stderr: stderr: 10+0 records in 2024-05-04T12:10:53.104 INFO:teuthology.orchestra.run.smithi079.stderr:10+0 records out 2024-05-04T12:10:53.104 INFO:teuthology.orchestra.run.smithi079.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0123693 s, 848 MB/s 2024-05-04T12:10:53.104 INFO:teuthology.orchestra.run.smithi079.stderr:--> Zapping successful for: 2024-05-04T12:10:53.268 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:53 smithi079 ceph-mon[27222]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-04T12:10:53.268 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:53 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:10:53.268 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:53 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:10:53.268 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:53 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi079", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-05-04T12:10:53.268 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:53 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-04T12:10:53.276 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:53 smithi146 ceph-mon[35311]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-04T12:10:53.277 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:53 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:10:53.277 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:53 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:10:53.277 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:53 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi079", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-05-04T12:10:53.277 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:53 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-04T12:10:54.209 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:54 smithi079 ceph-mon[27222]: Reconfiguring crash.smithi079 (monmap changed)... 2024-05-04T12:10:54.209 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:54 smithi079 ceph-mon[27222]: Reconfiguring daemon crash.smithi079 on smithi079 2024-05-04T12:10:54.209 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:54 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:10:54.209 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:54 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:10:54.277 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:54 smithi146 ceph-mon[35311]: Reconfiguring crash.smithi079 (monmap changed)... 2024-05-04T12:10:54.277 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:54 smithi146 ceph-mon[35311]: Reconfiguring daemon crash.smithi079 on smithi079 2024-05-04T12:10:54.277 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:54 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:10:54.277 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:54 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:10:54.497 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d9b985c4-0a0e-11ef-bc97-c7b262605968 -- ceph orch daemon add osd smithi079:vg_nvme/lv_4 2024-05-04T12:10:55.195 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:55 smithi079 ceph-mon[27222]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-04T12:10:55.195 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:55 smithi079 ceph-mon[27222]: Reconfiguring alertmanager.smithi079 (dependencies changed)... 2024-05-04T12:10:55.195 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:55 smithi079 ceph-mon[27222]: Reconfiguring daemon alertmanager.smithi079 on smithi079 2024-05-04T12:10:55.276 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:55 smithi146 ceph-mon[35311]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-04T12:10:55.277 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:55 smithi146 ceph-mon[35311]: Reconfiguring alertmanager.smithi079 (dependencies changed)... 2024-05-04T12:10:55.277 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:55 smithi146 ceph-mon[35311]: Reconfiguring daemon alertmanager.smithi079 on smithi079 2024-05-04T12:10:56.969 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/d9b985c4-0a0e-11ef-bc97-c7b262605968/mon.smithi079/config 2024-05-04T12:10:57.276 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:57 smithi146 ceph-mon[35311]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-04T12:10:57.276 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:57 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-04T12:10:57.315 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:57 smithi079 ceph-mon[27222]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-04T12:10:57.316 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:57 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-04T12:10:58.503 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:10:58.502+0000 7ff755582700 1 -- 172.21.15.79:0/1574508702 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff7500ffbb0 msgr2=0x7ff7500fffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:10:58.503 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:10:58.502+0000 7ff755582700 1 --2- 172.21.15.79:0/1574508702 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff7500ffbb0 0x7ff7500fffd0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7ff740009910 tx=0x7ff740009c20 comp rx=0 tx=0).stop 2024-05-04T12:10:58.503 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:10:58.502+0000 7ff755582700 1 -- 172.21.15.79:0/1574508702 shutdown_connections 2024-05-04T12:10:58.503 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:10:58.502+0000 7ff755582700 1 --2- 172.21.15.79:0/1574508702 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff750100db0 0x7ff750101210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:10:58.503 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:10:58.502+0000 7ff755582700 1 --2- 172.21.15.79:0/1574508702 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff7500ffbb0 0x7ff7500fffd0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:10:58.504 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:10:58.502+0000 7ff755582700 1 -- 172.21.15.79:0/1574508702 >> 172.21.15.79:0/1574508702 conn(0x7ff7500fb150 msgr2=0x7ff7500fd590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:10:58.504 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:10:58.502+0000 7ff755582700 1 -- 172.21.15.79:0/1574508702 shutdown_connections 2024-05-04T12:10:58.504 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:10:58.502+0000 7ff755582700 1 -- 172.21.15.79:0/1574508702 wait complete. 2024-05-04T12:10:58.504 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:10:58.503+0000 7ff755582700 1 Processor -- start 2024-05-04T12:10:58.504 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:10:58.503+0000 7ff755582700 1 -- start start 2024-05-04T12:10:58.504 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:10:58.503+0000 7ff755582700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff7500ffbb0 0x7ff750196540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:10:58.504 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:10:58.503+0000 7ff755582700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff750100db0 0x7ff750196a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:10:58.504 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:10:58.503+0000 7ff755582700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7501970a0 con 0x7ff750100db0 2024-05-04T12:10:58.504 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:10:58.503+0000 7ff755582700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7501971e0 con 0x7ff7500ffbb0 2024-05-04T12:10:58.504 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:10:58.503+0000 7ff74e7fc700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff750100db0 0x7ff750196a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:10:58.504 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:10:58.503+0000 7ff74e7fc700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff750100db0 0x7ff750196a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:45766/0 (socket says 172.21.15.79:45766) 2024-05-04T12:10:58.504 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:10:58.503+0000 7ff74e7fc700 1 -- 172.21.15.79:0/3686348392 learned_addr learned my addr 172.21.15.79:0/3686348392 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:10:58.505 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:10:58.503+0000 7ff74effd700 1 --2- 172.21.15.79:0/3686348392 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff7500ffbb0 0x7ff750196540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:10:58.505 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:10:58.503+0000 7ff74e7fc700 1 -- 172.21.15.79:0/3686348392 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff7500ffbb0 msgr2=0x7ff750196540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:10:58.505 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:10:58.503+0000 7ff74e7fc700 1 --2- 172.21.15.79:0/3686348392 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff7500ffbb0 0x7ff750196540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:10:58.505 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:10:58.503+0000 7ff74e7fc700 1 -- 172.21.15.79:0/3686348392 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7400095f0 con 0x7ff750100db0 2024-05-04T12:10:58.505 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:10:58.504+0000 7ff74e7fc700 1 --2- 172.21.15.79:0/3686348392 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff750100db0 0x7ff750196a80 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7ff74400b4b0 tx=0x7ff74400b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:10:58.505 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:10:58.504+0000 7ff737fff700 1 -- 172.21.15.79:0/3686348392 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff744011800 con 0x7ff750100db0 2024-05-04T12:10:58.505 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:10:58.504+0000 7ff755582700 1 -- 172.21.15.79:0/3686348392 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff75019bc90 con 0x7ff750100db0 2024-05-04T12:10:58.505 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:10:58.504+0000 7ff737fff700 1 -- 172.21.15.79:0/3686348392 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7ff744011e10 con 0x7ff750100db0 2024-05-04T12:10:58.505 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:10:58.504+0000 7ff737fff700 1 -- 172.21.15.79:0/3686348392 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff74400f2e0 con 0x7ff750100db0 2024-05-04T12:10:58.505 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:10:58.504+0000 7ff755582700 1 -- 172.21.15.79:0/3686348392 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff75019c1e0 con 0x7ff750100db0 2024-05-04T12:10:58.506 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:10:58.505+0000 7ff755582700 1 -- 172.21.15.79:0/3686348392 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff750108790 con 0x7ff750100db0 2024-05-04T12:10:58.506 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:10:58.505+0000 7ff737fff700 1 -- 172.21.15.79:0/3686348392 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 18) v1 ==== 90256+0+0 (secure 0 0 0) 0x7ff744010380 con 0x7ff750100db0 2024-05-04T12:10:58.507 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:10:58.506+0000 7ff737fff700 1 --2- 172.21.15.79:0/3686348392 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7ff73806c460 0x7ff73806e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:10:58.507 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:10:58.506+0000 7ff737fff700 1 -- 172.21.15.79:0/3686348392 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7ff74408a320 con 0x7ff750100db0 2024-05-04T12:10:58.507 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:10:58.506+0000 7ff74effd700 1 --2- 172.21.15.79:0/3686348392 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7ff73806c460 0x7ff73806e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:10:58.507 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:10:58.506+0000 7ff74effd700 1 --2- 172.21.15.79:0/3686348392 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7ff73806c460 0x7ff73806e920 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ff740006010 tx=0x7ff740005920 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:10:58.513 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:10:58.514+0000 7ff737fff700 1 -- 172.21.15.79:0/3686348392 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7ff744059670 con 0x7ff750100db0 2024-05-04T12:10:58.716 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:10:58.717+0000 7ff755582700 1 -- 172.21.15.79:0/3686348392 --> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi079:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7ff750069520 con 0x7ff73806c460 2024-05-04T12:10:59.276 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:59 smithi146 ceph-mon[35311]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-04T12:10:59.277 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:59 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-04T12:10:59.277 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:59 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-04T12:10:59.277 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:10:59 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-04T12:10:59.379 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:59 smithi079 ceph-mon[27222]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-04T12:10:59.379 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:59 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-04T12:10:59.380 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:59 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-04T12:10:59.380 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:10:59 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-04T12:11:00.276 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:00 smithi146 ceph-mon[35311]: from='client.14254 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi079:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-04T12:11:00.302 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:00 smithi079 ceph-mon[27222]: from='client.14254 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi079:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-04T12:11:01.195 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:01 smithi079 ceph-mon[27222]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-04T12:11:01.276 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:01 smithi146 ceph-mon[35311]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-04T12:11:02.695 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:02 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:11:02.695 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:02 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:11:02.695 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:02 smithi079 ceph-mon[27222]: Reconfiguring grafana.smithi079 (dependencies changed)... 2024-05-04T12:11:02.695 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:02 smithi079 ceph-mon[27222]: Reconfiguring daemon grafana.smithi079 on smithi079 2024-05-04T12:11:02.695 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:02 smithi079 ceph-mon[27222]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-04T12:11:02.776 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:02 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:11:02.776 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:02 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:11:02.777 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:02 smithi146 ceph-mon[35311]: Reconfiguring grafana.smithi079 (dependencies changed)... 2024-05-04T12:11:02.777 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:02 smithi146 ceph-mon[35311]: Reconfiguring daemon grafana.smithi079 on smithi079 2024-05-04T12:11:02.777 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:02 smithi146 ceph-mon[35311]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-04T12:11:03.695 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:03 smithi079 ceph-mon[27222]: from='client.? 172.21.15.79:0/4178779448' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "904ab939-d572-43db-8c6f-fb309334b478"}]: dispatch 2024-05-04T12:11:03.695 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:03 smithi079 ceph-mon[27222]: from='client.? 172.21.15.79:0/4178779448' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "904ab939-d572-43db-8c6f-fb309334b478"}]': finished 2024-05-04T12:11:03.695 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:03 smithi079 ceph-mon[27222]: osdmap e6: 1 total, 0 up, 1 in 2024-05-04T12:11:03.695 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:03 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-04T12:11:03.695 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:03 smithi079 ceph-mon[27222]: from='client.? 172.21.15.79:0/4262323989' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-04T12:11:03.776 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:03 smithi146 ceph-mon[35311]: from='client.? 172.21.15.79:0/4178779448' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "904ab939-d572-43db-8c6f-fb309334b478"}]: dispatch 2024-05-04T12:11:03.777 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:03 smithi146 ceph-mon[35311]: from='client.? 172.21.15.79:0/4178779448' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "904ab939-d572-43db-8c6f-fb309334b478"}]': finished 2024-05-04T12:11:03.777 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:03 smithi146 ceph-mon[35311]: osdmap e6: 1 total, 0 up, 1 in 2024-05-04T12:11:03.777 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:03 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-04T12:11:03.777 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:03 smithi146 ceph-mon[35311]: from='client.? 172.21.15.79:0/4262323989' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-04T12:11:04.694 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:04 smithi079 ceph-mon[27222]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-04T12:11:04.776 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:04 smithi146 ceph-mon[35311]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-04T12:11:07.194 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:07 smithi079 ceph-mon[27222]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-04T12:11:07.276 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:07 smithi146 ceph-mon[35311]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-04T12:11:09.271 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:09 smithi079 ceph-mon[27222]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-04T12:11:09.276 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:09 smithi146 ceph-mon[35311]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-04T12:11:11.195 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:11 smithi079 ceph-mon[27222]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-04T12:11:11.276 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:11 smithi146 ceph-mon[35311]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-04T12:11:12.681 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:12 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:11:12.681 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:12 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:11:12.681 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:12 smithi079 ceph-mon[27222]: Reconfiguring prometheus.smithi079 (dependencies changed)... 2024-05-04T12:11:12.681 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:12 smithi079 ceph-mon[27222]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-04T12:11:12.681 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:12 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-04T12:11:12.681 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:12 smithi079 ceph-mon[27222]: Reconfiguring daemon prometheus.smithi079 on smithi079 2024-05-04T12:11:12.776 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:12 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:11:12.777 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:12 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:11:12.777 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:12 smithi146 ceph-mon[35311]: Reconfiguring prometheus.smithi079 (dependencies changed)... 2024-05-04T12:11:12.777 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:12 smithi146 ceph-mon[35311]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-04T12:11:12.777 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:12 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-04T12:11:12.777 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:12 smithi146 ceph-mon[35311]: Reconfiguring daemon prometheus.smithi079 on smithi079 2024-05-04T12:11:13.776 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:13 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-05-04T12:11:13.776 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:13 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-04T12:11:13.777 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:13 smithi146 ceph-mon[35311]: Deploying daemon osd.0 on smithi079 2024-05-04T12:11:13.831 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:13 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-05-04T12:11:13.831 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:13 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-04T12:11:13.831 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:13 smithi079 ceph-mon[27222]: Deploying daemon osd.0 on smithi079 2024-05-04T12:11:14.670 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:14 smithi079 ceph-mon[27222]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-04T12:11:14.776 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:14 smithi146 ceph-mon[35311]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-04T12:11:17.268 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:17 smithi079 ceph-mon[27222]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-04T12:11:17.276 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:17 smithi146 ceph-mon[35311]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-04T12:11:18.275 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:18 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:11:18.275 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:18 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:11:18.275 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:18 smithi079 ceph-mon[27222]: Reconfiguring ceph-exporter.smithi146 (monmap changed)... 2024-05-04T12:11:18.275 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:18 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi146", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-05-04T12:11:18.275 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:18 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-04T12:11:18.276 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:18 smithi079 ceph-mon[27222]: Reconfiguring daemon ceph-exporter.smithi146 on smithi146 2024-05-04T12:11:18.276 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:18 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:11:18.276 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:18 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:11:18.277 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:18 smithi146 ceph-mon[35311]: Reconfiguring ceph-exporter.smithi146 (monmap changed)... 2024-05-04T12:11:18.277 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:18 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi146", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-05-04T12:11:18.277 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:18 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-04T12:11:18.277 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:18 smithi146 ceph-mon[35311]: Reconfiguring daemon ceph-exporter.smithi146 on smithi146 2024-05-04T12:11:19.195 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:19 smithi079 ceph-mon[27222]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-04T12:11:19.195 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:19 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:11:19.195 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:19 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:11:19.195 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:19 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi146", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-05-04T12:11:19.195 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:19 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-04T12:11:19.276 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:19 smithi146 ceph-mon[35311]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-04T12:11:19.276 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:19 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:11:19.277 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:19 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:11:19.277 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:19 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi146", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-05-04T12:11:19.277 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:19 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-04T12:11:20.236 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:20 smithi079 ceph-mon[27222]: Reconfiguring crash.smithi146 (monmap changed)... 2024-05-04T12:11:20.236 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:20 smithi079 ceph-mon[27222]: Reconfiguring daemon crash.smithi146 on smithi146 2024-05-04T12:11:20.236 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:20 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:11:20.236 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:20 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:11:20.236 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:20 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi146.eubqih", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-04T12:11:20.236 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:20 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-04T12:11:20.236 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:20 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-04T12:11:20.276 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:20 smithi146 ceph-mon[35311]: Reconfiguring crash.smithi146 (monmap changed)... 2024-05-04T12:11:20.276 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:20 smithi146 ceph-mon[35311]: Reconfiguring daemon crash.smithi146 on smithi146 2024-05-04T12:11:20.277 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:20 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:11:20.277 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:20 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:11:20.277 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:20 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi146.eubqih", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-04T12:11:20.277 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:20 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-04T12:11:20.277 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:20 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-04T12:11:21.195 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:21 smithi079 ceph-mon[27222]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-04T12:11:21.195 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:21 smithi079 ceph-mon[27222]: Reconfiguring mgr.smithi146.eubqih (monmap changed)... 2024-05-04T12:11:21.195 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:21 smithi079 ceph-mon[27222]: Reconfiguring daemon mgr.smithi146.eubqih on smithi146 2024-05-04T12:11:21.526 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:21 smithi146 ceph-mon[35311]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-04T12:11:21.526 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:21 smithi146 ceph-mon[35311]: Reconfiguring mgr.smithi146.eubqih (monmap changed)... 2024-05-04T12:11:21.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:21 smithi146 ceph-mon[35311]: Reconfiguring daemon mgr.smithi146.eubqih on smithi146 2024-05-04T12:11:22.695 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:22 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:11:22.695 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:22 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:11:22.695 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:22 smithi079 ceph-mon[27222]: Reconfiguring mon.smithi146 (monmap changed)... 2024-05-04T12:11:22.695 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:22 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-04T12:11:22.695 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:22 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-04T12:11:22.695 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:22 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-04T12:11:22.695 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:22 smithi079 ceph-mon[27222]: Reconfiguring daemon mon.smithi146 on smithi146 2024-05-04T12:11:22.696 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:22 smithi079 ceph-mon[27222]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-04T12:11:22.696 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:22 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:11:22.696 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:22 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:11:22.718 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:22 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:11:22.718 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:22 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:11:22.718 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:22 smithi146 ceph-mon[35311]: Reconfiguring mon.smithi146 (monmap changed)... 2024-05-04T12:11:22.718 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:22 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-04T12:11:22.718 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:22 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-04T12:11:22.719 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:22 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-04T12:11:22.719 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:22 smithi146 ceph-mon[35311]: Reconfiguring daemon mon.smithi146 on smithi146 2024-05-04T12:11:22.719 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:22 smithi146 ceph-mon[35311]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-04T12:11:22.719 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:22 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:11:22.719 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:22 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:11:23.883 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:23 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:11:23.883 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:23 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:11:23.883 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:23 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-05-04T12:11:23.883 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:23 smithi079 ceph-mon[27222]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-05-04T12:11:23.883 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:23 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi079:9093"}]: dispatch 2024-05-04T12:11:23.883 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:23 smithi079 ceph-mon[27222]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi079:9093"}]: dispatch 2024-05-04T12:11:23.883 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:23 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:11:23.883 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:23 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-05-04T12:11:23.883 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:23 smithi079 ceph-mon[27222]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-05-04T12:11:23.883 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:23 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi079:3000"}]: dispatch 2024-05-04T12:11:23.883 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:23 smithi079 ceph-mon[27222]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi079:3000"}]: dispatch 2024-05-04T12:11:23.884 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:23 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:11:23.884 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:23 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-05-04T12:11:23.884 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:23 smithi079 ceph-mon[27222]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-05-04T12:11:23.884 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:23 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi079:9095"}]: dispatch 2024-05-04T12:11:23.884 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:23 smithi079 ceph-mon[27222]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi079:9095"}]: dispatch 2024-05-04T12:11:23.884 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:23 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:11:23.884 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:23 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-04T12:11:24.027 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:23 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:11:24.027 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:23 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:11:24.027 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:23 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-05-04T12:11:24.027 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:23 smithi146 ceph-mon[35311]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-05-04T12:11:24.027 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:23 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi079:9093"}]: dispatch 2024-05-04T12:11:24.027 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:23 smithi146 ceph-mon[35311]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi079:9093"}]: dispatch 2024-05-04T12:11:24.027 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:23 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:11:24.027 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:23 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-05-04T12:11:24.027 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:23 smithi146 ceph-mon[35311]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-05-04T12:11:24.028 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:23 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi079:3000"}]: dispatch 2024-05-04T12:11:24.028 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:23 smithi146 ceph-mon[35311]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi079:3000"}]: dispatch 2024-05-04T12:11:24.028 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:23 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:11:24.028 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:23 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-05-04T12:11:24.028 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:23 smithi146 ceph-mon[35311]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-05-04T12:11:24.028 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:23 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi079:9095"}]: dispatch 2024-05-04T12:11:24.028 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:23 smithi146 ceph-mon[35311]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi079:9095"}]: dispatch 2024-05-04T12:11:24.028 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:23 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:11:24.028 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:23 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-04T12:11:25.151 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:24 smithi079 ceph-mon[27222]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-04T12:11:25.151 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:24 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:11:25.151 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:24 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:11:25.276 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:24 smithi146 ceph-mon[35311]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-04T12:11:25.277 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:24 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:11:25.277 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:24 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:11:25.537 INFO:teuthology.orchestra.run.smithi079.stdout:Created osd(s) 0 on host 'smithi079' 2024-05-04T12:11:25.537 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:11:25.538+0000 7ff737fff700 1 -- 172.21.15.79:0/3686348392 <== mgr.14219 v2:172.21.15.79:6800/402764217 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7ff750069520 con 0x7ff73806c460 2024-05-04T12:11:25.539 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:11:25.540+0000 7ff755582700 1 -- 172.21.15.79:0/3686348392 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7ff73806c460 msgr2=0x7ff73806e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:11:25.539 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:11:25.540+0000 7ff755582700 1 --2- 172.21.15.79:0/3686348392 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7ff73806c460 0x7ff73806e920 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ff740006010 tx=0x7ff740005920 comp rx=0 tx=0).stop 2024-05-04T12:11:25.540 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:11:25.540+0000 7ff755582700 1 -- 172.21.15.79:0/3686348392 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff750100db0 msgr2=0x7ff750196a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:11:25.540 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:11:25.540+0000 7ff755582700 1 --2- 172.21.15.79:0/3686348392 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff750100db0 0x7ff750196a80 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7ff74400b4b0 tx=0x7ff74400b870 comp rx=0 tx=0).stop 2024-05-04T12:11:25.540 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:11:25.540+0000 7ff755582700 1 -- 172.21.15.79:0/3686348392 shutdown_connections 2024-05-04T12:11:25.540 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:11:25.540+0000 7ff755582700 1 --2- 172.21.15.79:0/3686348392 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7ff73806c460 0x7ff73806e920 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:11:25.540 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:11:25.540+0000 7ff755582700 1 --2- 172.21.15.79:0/3686348392 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff7500ffbb0 0x7ff750196540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:11:25.540 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:11:25.540+0000 7ff755582700 1 --2- 172.21.15.79:0/3686348392 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff750100db0 0x7ff750196a80 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:11:25.540 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:11:25.541+0000 7ff755582700 1 -- 172.21.15.79:0/3686348392 >> 172.21.15.79:0/3686348392 conn(0x7ff7500fb150 msgr2=0x7ff750068e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:11:25.540 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:11:25.541+0000 7ff755582700 1 -- 172.21.15.79:0/3686348392 shutdown_connections 2024-05-04T12:11:25.540 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:11:25.541+0000 7ff755582700 1 -- 172.21.15.79:0/3686348392 wait complete. 2024-05-04T12:11:26.159 DEBUG:teuthology.orchestra.run.smithi079:osd.0> sudo journalctl -f -n 0 -u ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968@osd.0.service 2024-05-04T12:11:26.161 INFO:tasks.cephadm:Deploying osd.1 on smithi079 with /dev/vg_nvme/lv_3... 2024-05-04T12:11:26.161 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d9b985c4-0a0e-11ef-bc97-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-05-04T12:11:26.776 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:26 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:11:26.777 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:26 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:11:26.777 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:26 smithi146 ceph-mon[35311]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-04T12:11:26.777 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:26 smithi146 ceph-mon[35311]: from='osd.0 [v2:172.21.15.79:6802/1309201220,v1:172.21.15.79:6803/1309201220]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-05-04T12:11:26.809 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:26 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:11:26.809 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:26 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:11:26.809 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:26 smithi079 ceph-mon[27222]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-04T12:11:26.809 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:26 smithi079 ceph-mon[27222]: from='osd.0 [v2:172.21.15.79:6802/1309201220,v1:172.21.15.79:6803/1309201220]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-05-04T12:11:27.659 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/d9b985c4-0a0e-11ef-bc97-c7b262605968/mon.smithi079/config 2024-05-04T12:11:27.777 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:27 smithi146 ceph-mon[35311]: from='osd.0 [v2:172.21.15.79:6802/1309201220,v1:172.21.15.79:6803/1309201220]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-05-04T12:11:27.777 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:27 smithi146 ceph-mon[35311]: osdmap e7: 1 total, 0 up, 1 in 2024-05-04T12:11:27.777 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:27 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-04T12:11:27.777 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:27 smithi146 ceph-mon[35311]: from='osd.0 [v2:172.21.15.79:6802/1309201220,v1:172.21.15.79:6803/1309201220]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2024-05-04T12:11:27.777 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:27 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-04T12:11:27.782 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:27 smithi079 ceph-mon[27222]: from='osd.0 [v2:172.21.15.79:6802/1309201220,v1:172.21.15.79:6803/1309201220]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-05-04T12:11:27.782 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:27 smithi079 ceph-mon[27222]: osdmap e7: 1 total, 0 up, 1 in 2024-05-04T12:11:27.782 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:27 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-04T12:11:27.782 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:27 smithi079 ceph-mon[27222]: from='osd.0 [v2:172.21.15.79:6802/1309201220,v1:172.21.15.79:6803/1309201220]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2024-05-04T12:11:27.782 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:27 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-04T12:11:27.782 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:11:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:11:27.512+0000 7f234d983700 -1 osd.0 0 waiting for initial osdmap 2024-05-04T12:11:27.782 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:11:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:11:27.515+0000 7f2345f71700 -1 osd.0 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-04T12:11:28.695 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:28 smithi079 ceph-mon[27222]: from='osd.0 [v2:172.21.15.79:6802/1309201220,v1:172.21.15.79:6803/1309201220]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]': finished 2024-05-04T12:11:28.695 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:28 smithi079 ceph-mon[27222]: osdmap e8: 1 total, 0 up, 1 in 2024-05-04T12:11:28.695 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:28 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-04T12:11:28.695 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:28 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-04T12:11:28.695 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:28 smithi079 ceph-mon[27222]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-04T12:11:28.776 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:28 smithi146 ceph-mon[35311]: from='osd.0 [v2:172.21.15.79:6802/1309201220,v1:172.21.15.79:6803/1309201220]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]': finished 2024-05-04T12:11:28.777 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:28 smithi146 ceph-mon[35311]: osdmap e8: 1 total, 0 up, 1 in 2024-05-04T12:11:28.777 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:28 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-04T12:11:28.777 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:28 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-04T12:11:28.777 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:28 smithi146 ceph-mon[35311]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-04T12:11:29.637 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:29 smithi079 ceph-mon[27222]: purged_snaps scrub starts 2024-05-04T12:11:29.638 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:29 smithi079 ceph-mon[27222]: purged_snaps scrub ok 2024-05-04T12:11:29.638 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:29 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-04T12:11:29.638 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:29 smithi079 ceph-mon[27222]: osd.0 [v2:172.21.15.79:6802/1309201220,v1:172.21.15.79:6803/1309201220] boot 2024-05-04T12:11:29.638 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:29 smithi079 ceph-mon[27222]: osdmap e9: 1 total, 1 up, 1 in 2024-05-04T12:11:29.638 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:29 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-04T12:11:29.776 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:29 smithi146 ceph-mon[35311]: purged_snaps scrub starts 2024-05-04T12:11:29.777 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:29 smithi146 ceph-mon[35311]: purged_snaps scrub ok 2024-05-04T12:11:29.777 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:29 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-04T12:11:29.777 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:29 smithi146 ceph-mon[35311]: osd.0 [v2:172.21.15.79:6802/1309201220,v1:172.21.15.79:6803/1309201220] boot 2024-05-04T12:11:29.777 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:29 smithi146 ceph-mon[35311]: osdmap e9: 1 total, 1 up, 1 in 2024-05-04T12:11:29.777 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:29 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-04T12:11:30.695 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:30 smithi079 ceph-mon[27222]: osdmap e10: 1 total, 1 up, 1 in 2024-05-04T12:11:30.695 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:30 smithi079 ceph-mon[27222]: pgmap v37: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-04T12:11:30.695 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:30 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:11:30.695 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:30 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:11:30.744 INFO:teuthology.orchestra.run.smithi079.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-05-04T12:11:30.744 INFO:teuthology.orchestra.run.smithi079.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-05-04T12:11:30.744 INFO:teuthology.orchestra.run.smithi079.stderr: stderr: 10+0 records in 2024-05-04T12:11:30.744 INFO:teuthology.orchestra.run.smithi079.stderr:10+0 records out 2024-05-04T12:11:30.744 INFO:teuthology.orchestra.run.smithi079.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0151766 s, 691 MB/s 2024-05-04T12:11:30.744 INFO:teuthology.orchestra.run.smithi079.stderr: stderr: 2024-05-04T12:11:30.744 INFO:teuthology.orchestra.run.smithi079.stderr:--> Zapping successful for: 2024-05-04T12:11:30.776 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:30 smithi146 ceph-mon[35311]: osdmap e10: 1 total, 1 up, 1 in 2024-05-04T12:11:30.776 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:30 smithi146 ceph-mon[35311]: pgmap v37: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-04T12:11:30.777 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:30 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:11:30.777 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:30 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:11:31.343 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d9b985c4-0a0e-11ef-bc97-c7b262605968 -- ceph orch daemon add osd smithi079:vg_nvme/lv_3 2024-05-04T12:11:32.519 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/d9b985c4-0a0e-11ef-bc97-c7b262605968/mon.smithi079/config 2024-05-04T12:11:33.276 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:33 smithi146 ceph-mon[35311]: pgmap v38: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-04T12:11:33.444 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:33 smithi079 ceph-mon[27222]: pgmap v38: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-04T12:11:35.184 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:11:35.184+0000 7f06fb8a8700 1 -- 172.21.15.79:0/3852038458 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f06f40ff320 msgr2=0x7f06f40ff740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:11:35.184 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:11:35.184+0000 7f06fb8a8700 1 --2- 172.21.15.79:0/3852038458 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f06f40ff320 0x7f06f40ff740 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f06e4009910 tx=0x7f06e4009c20 comp rx=0 tx=0).stop 2024-05-04T12:11:35.185 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:11:35.184+0000 7f06fb8a8700 1 -- 172.21.15.79:0/3852038458 shutdown_connections 2024-05-04T12:11:35.185 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:11:35.184+0000 7f06fb8a8700 1 --2- 172.21.15.79:0/3852038458 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f06f4100520 0x7f06f4100980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:11:35.185 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:11:35.184+0000 7f06fb8a8700 1 --2- 172.21.15.79:0/3852038458 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f06f40ff320 0x7f06f40ff740 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:11:35.185 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:11:35.184+0000 7f06fb8a8700 1 -- 172.21.15.79:0/3852038458 >> 172.21.15.79:0/3852038458 conn(0x7f06f40fa8c0 msgr2=0x7f06f40fcd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:11:35.185 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:11:35.184+0000 7f06fb8a8700 1 -- 172.21.15.79:0/3852038458 shutdown_connections 2024-05-04T12:11:35.185 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:11:35.184+0000 7f06fb8a8700 1 -- 172.21.15.79:0/3852038458 wait complete. 2024-05-04T12:11:35.185 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:11:35.185+0000 7f06fb8a8700 1 Processor -- start 2024-05-04T12:11:35.185 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:11:35.185+0000 7f06fb8a8700 1 -- start start 2024-05-04T12:11:35.185 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:11:35.185+0000 7f06fb8a8700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f06f40ff320 0x7f06f4195400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:11:35.185 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:11:35.185+0000 7f06fb8a8700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f06f4100520 0x7f06f4195940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:11:35.185 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:11:35.185+0000 7f06fb8a8700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06f4195f60 con 0x7f06f40ff320 2024-05-04T12:11:35.185 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:11:35.185+0000 7f06fb8a8700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06f41960a0 con 0x7f06f4100520 2024-05-04T12:11:35.185 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:11:35.185+0000 7f06f9644700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f06f40ff320 0x7f06f4195400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:11:35.186 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:11:35.185+0000 7f06f9644700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f06f40ff320 0x7f06f4195400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:54778/0 (socket says 172.21.15.79:54778) 2024-05-04T12:11:35.186 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:11:35.185+0000 7f06f9644700 1 -- 172.21.15.79:0/1928296443 learned_addr learned my addr 172.21.15.79:0/1928296443 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:11:35.186 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:11:35.185+0000 7f06f8e43700 1 --2- 172.21.15.79:0/1928296443 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f06f4100520 0x7f06f4195940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:11:35.186 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:11:35.185+0000 7f06f9644700 1 -- 172.21.15.79:0/1928296443 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f06f4100520 msgr2=0x7f06f4195940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:11:35.186 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:11:35.185+0000 7f06f9644700 1 --2- 172.21.15.79:0/1928296443 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f06f4100520 0x7f06f4195940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:11:35.186 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:11:35.185+0000 7f06f9644700 1 -- 172.21.15.79:0/1928296443 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06e40095f0 con 0x7f06f40ff320 2024-05-04T12:11:35.186 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:11:35.185+0000 7f06f8e43700 1 --2- 172.21.15.79:0/1928296443 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f06f4100520 0x7f06f4195940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-04T12:11:35.186 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:11:35.185+0000 7f06f9644700 1 --2- 172.21.15.79:0/1928296443 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f06f40ff320 0x7f06f4195400 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f06e4000c00 tx=0x7f06e400f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:11:35.186 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:11:35.186+0000 7f06ea7fc700 1 -- 172.21.15.79:0/1928296443 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f06e401d070 con 0x7f06f40ff320 2024-05-04T12:11:35.186 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:11:35.186+0000 7f06ea7fc700 1 -- 172.21.15.79:0/1928296443 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f06e4005dc0 con 0x7f06f40ff320 2024-05-04T12:11:35.186 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:11:35.186+0000 7f06fb8a8700 1 -- 172.21.15.79:0/1928296443 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06f419aaf0 con 0x7f06f40ff320 2024-05-04T12:11:35.186 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:11:35.186+0000 7f06ea7fc700 1 -- 172.21.15.79:0/1928296443 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f06e4022470 con 0x7f06f40ff320 2024-05-04T12:11:35.186 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:11:35.186+0000 7f06fb8a8700 1 -- 172.21.15.79:0/1928296443 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06f419afe0 con 0x7f06f40ff320 2024-05-04T12:11:35.187 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:11:35.187+0000 7f06fb8a8700 1 -- 172.21.15.79:0/1928296443 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06f4108720 con 0x7f06f40ff320 2024-05-04T12:11:35.187 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:11:35.188+0000 7f06ea7fc700 1 -- 172.21.15.79:0/1928296443 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 18) v1 ==== 90256+0+0 (secure 0 0 0) 0x7f06e4003810 con 0x7f06f40ff320 2024-05-04T12:11:35.188 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:11:35.188+0000 7f06ea7fc700 1 --2- 172.21.15.79:0/1928296443 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f06e006c460 0x7f06e006e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:11:35.188 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:11:35.188+0000 7f06ea7fc700 1 -- 172.21.15.79:0/1928296443 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1982+0+0 (secure 0 0 0) 0x7f06e408be60 con 0x7f06f40ff320 2024-05-04T12:11:35.188 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:11:35.188+0000 7f06f8e43700 1 --2- 172.21.15.79:0/1928296443 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f06e006c460 0x7f06e006e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:11:35.188 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:11:35.189+0000 7f06f8e43700 1 --2- 172.21.15.79:0/1928296443 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f06e006c460 0x7f06e006e920 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f06f0011490 tx=0x7f06f0009260 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:11:35.191 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:11:35.193+0000 7f06ea7fc700 1 -- 172.21.15.79:0/1928296443 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f06e405af50 con 0x7f06f40ff320 2024-05-04T12:11:35.276 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:35 smithi146 ceph-mon[35311]: pgmap v39: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-04T12:11:35.276 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:35 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:11:35.276 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:35 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:11:35.277 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:35 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-04T12:11:35.277 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:35 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:11:35.277 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:35 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-04T12:11:35.277 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:35 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-04T12:11:35.277 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:35 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:11:35.277 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:35 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-04T12:11:35.375 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:35 smithi079 ceph-mon[27222]: pgmap v39: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-04T12:11:35.375 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:35 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:11:35.375 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:35 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:11:35.375 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:35 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-04T12:11:35.375 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:35 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:11:35.375 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:35 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-04T12:11:35.375 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:35 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-04T12:11:35.375 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:35 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:11:35.376 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:35 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-04T12:11:35.417 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:11:35.417+0000 7f06fb8a8700 1 -- 172.21.15.79:0/1928296443 --> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi079:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f06f419b620 con 0x7f06e006c460 2024-05-04T12:11:36.193 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:36 smithi079 ceph-mon[27222]: Detected new or changed devices on smithi079 2024-05-04T12:11:36.194 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:36 smithi079 ceph-mon[27222]: Adjusting osd_memory_target on smithi079 to 11840M 2024-05-04T12:11:36.194 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:36 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-04T12:11:36.194 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:36 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-04T12:11:36.194 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:36 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-04T12:11:36.276 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:36 smithi146 ceph-mon[35311]: Detected new or changed devices on smithi079 2024-05-04T12:11:36.277 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:36 smithi146 ceph-mon[35311]: Adjusting osd_memory_target on smithi079 to 11840M 2024-05-04T12:11:36.277 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:36 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-04T12:11:36.277 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:36 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-04T12:11:36.277 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:36 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-04T12:11:37.017 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:37 smithi146 ceph-mon[35311]: from='client.14272 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi079:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-04T12:11:37.018 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:37 smithi146 ceph-mon[35311]: pgmap v40: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-04T12:11:37.018 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:37 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:11:37.018 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:37 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:11:37.127 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:37 smithi079 ceph-mon[27222]: from='client.14272 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi079:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-04T12:11:37.127 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:37 smithi079 ceph-mon[27222]: pgmap v40: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-04T12:11:37.127 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:37 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:11:37.127 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:37 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:11:39.195 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:39 smithi079 ceph-mon[27222]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-04T12:11:39.195 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:39 smithi079 ceph-mon[27222]: from='client.? 172.21.15.79:0/358584349' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bce71499-a9a8-4d8f-932e-1c85918eee61"}]: dispatch 2024-05-04T12:11:39.195 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:39 smithi079 ceph-mon[27222]: from='client.? 172.21.15.79:0/358584349' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bce71499-a9a8-4d8f-932e-1c85918eee61"}]': finished 2024-05-04T12:11:39.195 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:39 smithi079 ceph-mon[27222]: osdmap e11: 2 total, 1 up, 2 in 2024-05-04T12:11:39.195 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:39 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-04T12:11:39.195 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:39 smithi079 ceph-mon[27222]: from='client.? 172.21.15.79:0/1442709328' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-04T12:11:39.526 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:39 smithi146 ceph-mon[35311]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-04T12:11:39.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:39 smithi146 ceph-mon[35311]: from='client.? 172.21.15.79:0/358584349' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bce71499-a9a8-4d8f-932e-1c85918eee61"}]: dispatch 2024-05-04T12:11:39.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:39 smithi146 ceph-mon[35311]: from='client.? 172.21.15.79:0/358584349' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bce71499-a9a8-4d8f-932e-1c85918eee61"}]': finished 2024-05-04T12:11:39.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:39 smithi146 ceph-mon[35311]: osdmap e11: 2 total, 1 up, 2 in 2024-05-04T12:11:39.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:39 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-04T12:11:39.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:39 smithi146 ceph-mon[35311]: from='client.? 172.21.15.79:0/1442709328' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-04T12:11:41.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:41 smithi079 ceph-mon[27222]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-04T12:11:41.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:41 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:11:41.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:41 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:11:41.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:41 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-04T12:11:41.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:41 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-04T12:11:41.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:41 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:11:41.526 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:41 smithi146 ceph-mon[35311]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-04T12:11:41.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:41 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:11:41.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:41 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:11:41.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:41 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-04T12:11:41.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:41 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-04T12:11:41.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:41 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:11:42.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:42 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-04T12:11:42.526 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:42 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-04T12:11:43.385 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:43 smithi079 ceph-mon[27222]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-04T12:11:43.526 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:43 smithi146 ceph-mon[35311]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-04T12:11:45.418 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:45 smithi079 ceph-mon[27222]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-04T12:11:45.526 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:45 smithi146 ceph-mon[35311]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-04T12:11:46.344 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:46 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-05-04T12:11:46.345 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:46 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-04T12:11:46.526 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:46 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-05-04T12:11:46.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:46 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-04T12:11:47.185 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:47 smithi079 ceph-mon[27222]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-04T12:11:47.186 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:47 smithi079 ceph-mon[27222]: Deploying daemon osd.1 on smithi079 2024-05-04T12:11:47.526 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:47 smithi146 ceph-mon[35311]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-04T12:11:47.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:47 smithi146 ceph-mon[35311]: Deploying daemon osd.1 on smithi079 2024-05-04T12:11:49.354 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:49 smithi079 ceph-mon[27222]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-04T12:11:49.526 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:49 smithi146 ceph-mon[35311]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-04T12:11:51.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:51 smithi079 ceph-mon[27222]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-04T12:11:51.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:51 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:11:51.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:51 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:11:51.526 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:51 smithi146 ceph-mon[35311]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-04T12:11:51.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:51 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:11:51.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:51 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:11:52.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:52 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-04T12:11:52.505 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:52 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-04T12:11:53.195 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:53 smithi079 ceph-mon[27222]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-04T12:11:53.195 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:53 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:11:53.195 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:53 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:11:53.526 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:53 smithi146 ceph-mon[35311]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-04T12:11:53.526 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:53 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:11:53.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:53 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:11:54.592 INFO:teuthology.orchestra.run.smithi079.stdout:Created osd(s) 1 on host 'smithi079' 2024-05-04T12:11:54.592 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:11:54.593+0000 7f06ea7fc700 1 -- 172.21.15.79:0/1928296443 <== mgr.14219 v2:172.21.15.79:6800/402764217 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f06f419b620 con 0x7f06e006c460 2024-05-04T12:11:54.594 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:11:54.595+0000 7f06fb8a8700 1 -- 172.21.15.79:0/1928296443 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f06e006c460 msgr2=0x7f06e006e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:11:54.594 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:11:54.595+0000 7f06fb8a8700 1 --2- 172.21.15.79:0/1928296443 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f06e006c460 0x7f06e006e920 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f06f0011490 tx=0x7f06f0009260 comp rx=0 tx=0).stop 2024-05-04T12:11:54.595 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:11:54.595+0000 7f06fb8a8700 1 -- 172.21.15.79:0/1928296443 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f06f40ff320 msgr2=0x7f06f4195400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:11:54.595 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:11:54.595+0000 7f06fb8a8700 1 --2- 172.21.15.79:0/1928296443 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f06f40ff320 0x7f06f4195400 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f06e4000c00 tx=0x7f06e400f690 comp rx=0 tx=0).stop 2024-05-04T12:11:54.595 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:11:54.596+0000 7f06fb8a8700 1 -- 172.21.15.79:0/1928296443 shutdown_connections 2024-05-04T12:11:54.595 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:11:54.596+0000 7f06fb8a8700 1 --2- 172.21.15.79:0/1928296443 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f06e006c460 0x7f06e006e920 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:11:54.595 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:11:54.596+0000 7f06fb8a8700 1 --2- 172.21.15.79:0/1928296443 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f06f40ff320 0x7f06f4195400 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:11:54.595 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:11:54.596+0000 7f06fb8a8700 1 --2- 172.21.15.79:0/1928296443 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f06f4100520 0x7f06f4195940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:11:54.595 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:11:54.596+0000 7f06fb8a8700 1 -- 172.21.15.79:0/1928296443 >> 172.21.15.79:0/1928296443 conn(0x7f06f40fa8c0 msgr2=0x7f06f4103750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:11:54.595 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:11:54.596+0000 7f06fb8a8700 1 -- 172.21.15.79:0/1928296443 shutdown_connections 2024-05-04T12:11:54.595 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:11:54.596+0000 7f06fb8a8700 1 -- 172.21.15.79:0/1928296443 wait complete. 2024-05-04T12:11:55.257 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:55 smithi079 ceph-mon[27222]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-04T12:11:55.258 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:55 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:11:55.258 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:55 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:11:55.258 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:55 smithi079 ceph-mon[27222]: from='osd.1 [v2:172.21.15.79:6810/3443379349,v1:172.21.15.79:6811/3443379349]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-04T12:11:55.526 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:55 smithi146 ceph-mon[35311]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-04T12:11:55.526 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:55 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:11:55.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:55 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:11:55.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:55 smithi146 ceph-mon[35311]: from='osd.1 [v2:172.21.15.79:6810/3443379349,v1:172.21.15.79:6811/3443379349]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-04T12:11:55.571 DEBUG:teuthology.orchestra.run.smithi079:osd.1> sudo journalctl -f -n 0 -u ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968@osd.1.service 2024-05-04T12:11:55.574 INFO:tasks.cephadm:Deploying osd.2 on smithi079 with /dev/vg_nvme/lv_2... 2024-05-04T12:11:55.574 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d9b985c4-0a0e-11ef-bc97-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-05-04T12:11:56.284 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/d9b985c4-0a0e-11ef-bc97-c7b262605968/mon.smithi079/config 2024-05-04T12:11:56.945 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:56 smithi079 ceph-mon[27222]: from='osd.1 [v2:172.21.15.79:6810/3443379349,v1:172.21.15.79:6811/3443379349]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-05-04T12:11:56.945 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:56 smithi079 ceph-mon[27222]: osdmap e12: 2 total, 1 up, 2 in 2024-05-04T12:11:56.945 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:56 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-04T12:11:56.945 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:56 smithi079 ceph-mon[27222]: from='osd.1 [v2:172.21.15.79:6810/3443379349,v1:172.21.15.79:6811/3443379349]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2024-05-04T12:11:56.945 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:56 smithi079 ceph-mon[27222]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-04T12:11:56.945 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:11:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:11:56.577+0000 7f4dacde8700 -1 osd.1 0 waiting for initial osdmap 2024-05-04T12:11:56.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:11:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:11:56.582+0000 7f4da93de700 -1 osd.1 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-04T12:11:57.026 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:56 smithi146 ceph-mon[35311]: from='osd.1 [v2:172.21.15.79:6810/3443379349,v1:172.21.15.79:6811/3443379349]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-05-04T12:11:57.026 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:56 smithi146 ceph-mon[35311]: osdmap e12: 2 total, 1 up, 2 in 2024-05-04T12:11:57.026 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:56 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-04T12:11:57.027 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:56 smithi146 ceph-mon[35311]: from='osd.1 [v2:172.21.15.79:6810/3443379349,v1:172.21.15.79:6811/3443379349]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2024-05-04T12:11:57.027 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:56 smithi146 ceph-mon[35311]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-04T12:11:57.695 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:57 smithi079 ceph-mon[27222]: from='osd.1 [v2:172.21.15.79:6810/3443379349,v1:172.21.15.79:6811/3443379349]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]': finished 2024-05-04T12:11:57.695 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:57 smithi079 ceph-mon[27222]: osdmap e13: 2 total, 1 up, 2 in 2024-05-04T12:11:57.695 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:57 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-04T12:11:57.695 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:57 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-04T12:11:57.695 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:57 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-04T12:11:58.026 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:57 smithi146 ceph-mon[35311]: from='osd.1 [v2:172.21.15.79:6810/3443379349,v1:172.21.15.79:6811/3443379349]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]': finished 2024-05-04T12:11:58.027 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:57 smithi146 ceph-mon[35311]: osdmap e13: 2 total, 1 up, 2 in 2024-05-04T12:11:58.027 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:57 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-04T12:11:58.027 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:57 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-04T12:11:58.027 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:57 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-04T12:11:58.929 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:58 smithi079 ceph-mon[27222]: purged_snaps scrub starts 2024-05-04T12:11:58.929 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:58 smithi079 ceph-mon[27222]: purged_snaps scrub ok 2024-05-04T12:11:58.929 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:58 smithi079 ceph-mon[27222]: osd.1 [v2:172.21.15.79:6810/3443379349,v1:172.21.15.79:6811/3443379349] boot 2024-05-04T12:11:58.929 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:58 smithi079 ceph-mon[27222]: osdmap e14: 2 total, 2 up, 2 in 2024-05-04T12:11:58.929 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:58 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-04T12:11:58.929 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:58 smithi079 ceph-mon[27222]: pgmap v55: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-04T12:11:59.026 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:58 smithi146 ceph-mon[35311]: purged_snaps scrub starts 2024-05-04T12:11:59.027 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:58 smithi146 ceph-mon[35311]: purged_snaps scrub ok 2024-05-04T12:11:59.027 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:58 smithi146 ceph-mon[35311]: osd.1 [v2:172.21.15.79:6810/3443379349,v1:172.21.15.79:6811/3443379349] boot 2024-05-04T12:11:59.027 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:58 smithi146 ceph-mon[35311]: osdmap e14: 2 total, 2 up, 2 in 2024-05-04T12:11:59.027 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:58 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-04T12:11:59.027 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:58 smithi146 ceph-mon[35311]: pgmap v55: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-04T12:11:59.821 INFO:teuthology.orchestra.run.smithi079.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-05-04T12:11:59.821 INFO:teuthology.orchestra.run.smithi079.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-05-04T12:11:59.821 INFO:teuthology.orchestra.run.smithi079.stderr: stderr: 10+0 records in 2024-05-04T12:11:59.821 INFO:teuthology.orchestra.run.smithi079.stderr:10+0 records out 2024-05-04T12:11:59.822 INFO:teuthology.orchestra.run.smithi079.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0128466 s, 816 MB/s 2024-05-04T12:11:59.822 INFO:teuthology.orchestra.run.smithi079.stderr:--> Zapping successful for: 2024-05-04T12:11:59.855 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:11:59 smithi079 ceph-mon[27222]: osdmap e15: 2 total, 2 up, 2 in 2024-05-04T12:12:00.026 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:11:59 smithi146 ceph-mon[35311]: osdmap e15: 2 total, 2 up, 2 in 2024-05-04T12:12:00.471 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d9b985c4-0a0e-11ef-bc97-c7b262605968 -- ceph orch daemon add osd smithi079:vg_nvme/lv_2 2024-05-04T12:12:00.751 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:00 smithi079 ceph-mon[27222]: pgmap v57: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-04T12:12:00.751 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:00 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:12:00.751 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:00 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:12:01.026 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:00 smithi146 ceph-mon[35311]: pgmap v57: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-04T12:12:01.027 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:00 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:12:01.027 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:00 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:12:01.179 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/d9b985c4-0a0e-11ef-bc97-c7b262605968/mon.smithi079/config 2024-05-04T12:12:03.032 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:03 smithi079 ceph-mon[27222]: pgmap v58: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-04T12:12:03.251 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:12:03.249+0000 7fd691710700 1 -- 172.21.15.79:0/2302692409 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd68c0fef80 msgr2=0x7fd68c0ff3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:12:03.251 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:12:03.249+0000 7fd691710700 1 --2- 172.21.15.79:0/2302692409 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd68c0fef80 0x7fd68c0ff3a0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fd67c009910 tx=0x7fd67c009c20 comp rx=0 tx=0).stop 2024-05-04T12:12:03.251 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:12:03.249+0000 7fd691710700 1 -- 172.21.15.79:0/2302692409 shutdown_connections 2024-05-04T12:12:03.251 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:12:03.249+0000 7fd691710700 1 --2- 172.21.15.79:0/2302692409 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd68c102000 0x7fd68c102460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:12:03.251 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:12:03.249+0000 7fd691710700 1 --2- 172.21.15.79:0/2302692409 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd68c0fef80 0x7fd68c0ff3a0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:12:03.251 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:12:03.249+0000 7fd691710700 1 -- 172.21.15.79:0/2302692409 >> 172.21.15.79:0/2302692409 conn(0x7fd68c0fa8c0 msgr2=0x7fd68c0fcd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:12:03.251 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:12:03.249+0000 7fd691710700 1 -- 172.21.15.79:0/2302692409 shutdown_connections 2024-05-04T12:12:03.251 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:12:03.249+0000 7fd691710700 1 -- 172.21.15.79:0/2302692409 wait complete. 2024-05-04T12:12:03.252 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:12:03.250+0000 7fd691710700 1 Processor -- start 2024-05-04T12:12:03.252 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:12:03.250+0000 7fd691710700 1 -- start start 2024-05-04T12:12:03.252 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:12:03.250+0000 7fd691710700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd68c0fef80 0x7fd68c101960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:12:03.252 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:12:03.250+0000 7fd691710700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd68c102000 0x7fd68c0fffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:12:03.252 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:12:03.250+0000 7fd691710700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd68c1004f0 con 0x7fd68c102000 2024-05-04T12:12:03.252 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:12:03.250+0000 7fd691710700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd68c100630 con 0x7fd68c0fef80 2024-05-04T12:12:03.252 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:12:03.250+0000 7fd68a7fc700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd68c102000 0x7fd68c0fffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:12:03.252 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:12:03.250+0000 7fd68a7fc700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd68c102000 0x7fd68c0fffb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:40770/0 (socket says 172.21.15.79:40770) 2024-05-04T12:12:03.252 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:12:03.250+0000 7fd68a7fc700 1 -- 172.21.15.79:0/134108232 learned_addr learned my addr 172.21.15.79:0/134108232 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:12:03.252 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:12:03.251+0000 7fd68affd700 1 --2- 172.21.15.79:0/134108232 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd68c0fef80 0x7fd68c101960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:12:03.252 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:12:03.251+0000 7fd68a7fc700 1 -- 172.21.15.79:0/134108232 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd68c0fef80 msgr2=0x7fd68c101960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:12:03.252 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:12:03.251+0000 7fd68a7fc700 1 --2- 172.21.15.79:0/134108232 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd68c0fef80 0x7fd68c101960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:12:03.252 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:12:03.251+0000 7fd68a7fc700 1 -- 172.21.15.79:0/134108232 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd67c0095f0 con 0x7fd68c102000 2024-05-04T12:12:03.252 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:12:03.251+0000 7fd68a7fc700 1 --2- 172.21.15.79:0/134108232 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd68c102000 0x7fd68c0fffb0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fd68000b4b0 tx=0x7fd68000b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:12:03.253 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:12:03.251+0000 7fd673fff700 1 -- 172.21.15.79:0/134108232 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd680011800 con 0x7fd68c102000 2024-05-04T12:12:03.253 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:12:03.251+0000 7fd673fff700 1 -- 172.21.15.79:0/134108232 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fd680011e10 con 0x7fd68c102000 2024-05-04T12:12:03.253 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:12:03.251+0000 7fd691710700 1 -- 172.21.15.79:0/134108232 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd68c100910 con 0x7fd68c102000 2024-05-04T12:12:03.253 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:12:03.251+0000 7fd673fff700 1 -- 172.21.15.79:0/134108232 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd68000f2e0 con 0x7fd68c102000 2024-05-04T12:12:03.253 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:12:03.251+0000 7fd691710700 1 -- 172.21.15.79:0/134108232 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd68c19e5e0 con 0x7fd68c102000 2024-05-04T12:12:03.253 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:12:03.252+0000 7fd691710700 1 -- 172.21.15.79:0/134108232 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd68c109880 con 0x7fd68c102000 2024-05-04T12:12:03.254 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:12:03.253+0000 7fd673fff700 1 -- 172.21.15.79:0/134108232 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 18) v1 ==== 90256+0+0 (secure 0 0 0) 0x7fd680010380 con 0x7fd68c102000 2024-05-04T12:12:03.254 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:12:03.253+0000 7fd673fff700 1 --2- 172.21.15.79:0/134108232 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7fd67406c460 0x7fd67406e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:12:03.254 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:12:03.253+0000 7fd673fff700 1 -- 172.21.15.79:0/134108232 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 2414+0+0 (secure 0 0 0) 0x7fd68008a690 con 0x7fd68c102000 2024-05-04T12:12:03.254 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:12:03.253+0000 7fd68affd700 1 --2- 172.21.15.79:0/134108232 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7fd67406c460 0x7fd67406e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:12:03.254 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:12:03.254+0000 7fd68affd700 1 --2- 172.21.15.79:0/134108232 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7fd67406c460 0x7fd67406e920 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fd67c006010 tx=0x7fd67c00b540 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:12:03.256 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:12:03.258+0000 7fd673fff700 1 -- 172.21.15.79:0/134108232 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fd6800595d0 con 0x7fd68c102000 2024-05-04T12:12:03.276 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:03 smithi146 ceph-mon[35311]: pgmap v58: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-04T12:12:03.462 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:12:03.463+0000 7fd691710700 1 -- 172.21.15.79:0/134108232 --> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi079:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7fd68c106950 con 0x7fd67406c460 2024-05-04T12:12:04.276 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:04 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-04T12:12:04.276 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:04 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-04T12:12:04.277 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:04 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-04T12:12:04.373 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:04 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-04T12:12:04.373 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:04 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-04T12:12:04.373 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:04 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-04T12:12:05.276 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:05 smithi146 ceph-mon[35311]: from='client.14290 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi079:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-04T12:12:05.277 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:05 smithi146 ceph-mon[35311]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-04T12:12:05.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:05 smithi079 ceph-mon[27222]: from='client.14290 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi079:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-04T12:12:05.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:05 smithi079 ceph-mon[27222]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-04T12:12:06.724 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:06 smithi146 ceph-mon[35311]: Detected new or changed devices on smithi079 2024-05-04T12:12:06.724 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:06 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:12:06.725 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:06 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:12:06.725 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:06 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-04T12:12:06.725 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:06 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-04T12:12:06.725 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:06 smithi146 ceph-mon[35311]: Adjusting osd_memory_target on smithi079 to 5920M 2024-05-04T12:12:06.725 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:06 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:12:06.725 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:06 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-04T12:12:06.725 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:06 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-04T12:12:06.725 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:06 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:12:06.725 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:06 smithi146 ceph-mon[35311]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-04T12:12:06.725 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:06 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-04T12:12:06.945 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:06 smithi079 ceph-mon[27222]: Detected new or changed devices on smithi079 2024-05-04T12:12:06.945 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:06 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:12:06.945 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:06 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:12:06.945 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:06 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-04T12:12:06.945 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:06 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-04T12:12:06.945 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:06 smithi079 ceph-mon[27222]: Adjusting osd_memory_target on smithi079 to 5920M 2024-05-04T12:12:06.945 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:06 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:12:06.946 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:06 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-04T12:12:06.946 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:06 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-04T12:12:06.946 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:06 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:12:06.946 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:06 smithi079 ceph-mon[27222]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-04T12:12:06.946 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:06 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-04T12:12:07.945 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:07 smithi079 ceph-mon[27222]: from='client.? 172.21.15.79:0/823192895' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cfb80418-73e3-4001-8d30-53f06365d435"}]: dispatch 2024-05-04T12:12:07.945 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:07 smithi079 ceph-mon[27222]: from='client.? 172.21.15.79:0/823192895' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cfb80418-73e3-4001-8d30-53f06365d435"}]': finished 2024-05-04T12:12:07.945 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:07 smithi079 ceph-mon[27222]: osdmap e16: 3 total, 2 up, 3 in 2024-05-04T12:12:07.945 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:07 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-04T12:12:07.945 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:07 smithi079 ceph-mon[27222]: from='client.? 172.21.15.79:0/3692586076' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-04T12:12:08.026 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:07 smithi146 ceph-mon[35311]: from='client.? 172.21.15.79:0/823192895' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cfb80418-73e3-4001-8d30-53f06365d435"}]: dispatch 2024-05-04T12:12:08.026 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:07 smithi146 ceph-mon[35311]: from='client.? 172.21.15.79:0/823192895' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cfb80418-73e3-4001-8d30-53f06365d435"}]': finished 2024-05-04T12:12:08.026 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:07 smithi146 ceph-mon[35311]: osdmap e16: 3 total, 2 up, 3 in 2024-05-04T12:12:08.026 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:07 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-04T12:12:08.027 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:07 smithi146 ceph-mon[35311]: from='client.? 172.21.15.79:0/3692586076' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-04T12:12:09.026 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:08 smithi146 ceph-mon[35311]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-04T12:12:09.026 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:08 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:12:09.027 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:08 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:12:09.194 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:08 smithi079 ceph-mon[27222]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-04T12:12:09.195 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:08 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:12:09.195 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:08 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:12:11.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:11 smithi079 ceph-mon[27222]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-04T12:12:11.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:11 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:12:11.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:11 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:12:11.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:11 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-04T12:12:11.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:11 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-04T12:12:11.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:11 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:12:11.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:11 smithi146 ceph-mon[35311]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-04T12:12:11.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:11 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:12:11.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:11 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:12:11.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:11 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-04T12:12:11.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:11 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-04T12:12:11.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:11 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:12:12.398 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:12 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-04T12:12:12.526 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:12 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-04T12:12:13.375 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:13 smithi079 ceph-mon[27222]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-04T12:12:13.526 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:13 smithi146 ceph-mon[35311]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-04T12:12:14.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:14 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-05-04T12:12:14.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:14 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-04T12:12:14.526 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:14 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-05-04T12:12:14.526 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:14 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-04T12:12:15.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:15 smithi079 ceph-mon[27222]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-04T12:12:15.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:15 smithi079 ceph-mon[27222]: Deploying daemon osd.2 on smithi079 2024-05-04T12:12:15.526 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:15 smithi146 ceph-mon[35311]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-04T12:12:15.526 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:15 smithi146 ceph-mon[35311]: Deploying daemon osd.2 on smithi079 2024-05-04T12:12:17.337 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:17 smithi079 ceph-mon[27222]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-04T12:12:17.526 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:17 smithi146 ceph-mon[35311]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-04T12:12:19.195 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:19 smithi079 ceph-mon[27222]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-04T12:12:19.526 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:19 smithi146 ceph-mon[35311]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-04T12:12:20.776 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:20 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:12:20.776 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:20 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-04T12:12:20.776 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:20 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:12:20.776 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:20 smithi146 ceph-mon[35311]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-04T12:12:20.944 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:20 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:12:20.945 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:20 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-04T12:12:20.945 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:20 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:12:20.945 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:20 smithi079 ceph-mon[27222]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-04T12:12:22.580 INFO:teuthology.orchestra.run.smithi079.stdout:Created osd(s) 2 on host 'smithi079' 2024-05-04T12:12:22.580 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:12:22.581+0000 7fd673fff700 1 -- 172.21.15.79:0/134108232 <== mgr.14219 v2:172.21.15.79:6800/402764217 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fd68c106950 con 0x7fd67406c460 2024-05-04T12:12:22.583 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:12:22.584+0000 7fd691710700 1 -- 172.21.15.79:0/134108232 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7fd67406c460 msgr2=0x7fd67406e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:12:22.583 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:12:22.584+0000 7fd691710700 1 --2- 172.21.15.79:0/134108232 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7fd67406c460 0x7fd67406e920 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fd67c006010 tx=0x7fd67c00b540 comp rx=0 tx=0).stop 2024-05-04T12:12:22.583 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:12:22.584+0000 7fd691710700 1 -- 172.21.15.79:0/134108232 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd68c102000 msgr2=0x7fd68c0fffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:12:22.583 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:12:22.584+0000 7fd691710700 1 --2- 172.21.15.79:0/134108232 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd68c102000 0x7fd68c0fffb0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fd68000b4b0 tx=0x7fd68000b7c0 comp rx=0 tx=0).stop 2024-05-04T12:12:22.583 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:12:22.584+0000 7fd691710700 1 -- 172.21.15.79:0/134108232 shutdown_connections 2024-05-04T12:12:22.583 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:12:22.584+0000 7fd691710700 1 --2- 172.21.15.79:0/134108232 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7fd67406c460 0x7fd67406e920 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:12:22.583 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:12:22.584+0000 7fd691710700 1 --2- 172.21.15.79:0/134108232 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd68c0fef80 0x7fd68c101960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:12:22.583 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:12:22.584+0000 7fd691710700 1 --2- 172.21.15.79:0/134108232 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd68c102000 0x7fd68c0fffb0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:12:22.583 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:12:22.584+0000 7fd691710700 1 -- 172.21.15.79:0/134108232 >> 172.21.15.79:0/134108232 conn(0x7fd68c0fa8c0 msgr2=0x7fd68c105230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:12:22.583 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:12:22.584+0000 7fd691710700 1 -- 172.21.15.79:0/134108232 shutdown_connections 2024-05-04T12:12:22.583 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:12:22.584+0000 7fd691710700 1 -- 172.21.15.79:0/134108232 wait complete. 2024-05-04T12:12:23.173 DEBUG:teuthology.orchestra.run.smithi079:osd.2> sudo journalctl -f -n 0 -u ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968@osd.2.service 2024-05-04T12:12:23.175 INFO:tasks.cephadm:Deploying osd.3 on smithi146 with /dev/vg_nvme/lv_4... 2024-05-04T12:12:23.175 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d9b985c4-0a0e-11ef-bc97-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-05-04T12:12:23.201 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:23 smithi146 ceph-mon[35311]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-04T12:12:23.231 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:23 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:12:23.232 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:23 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:12:23.318 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:23 smithi079 ceph-mon[27222]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-04T12:12:23.318 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:23 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:12:23.318 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:23 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:12:23.384 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/d9b985c4-0a0e-11ef-bc97-c7b262605968/mon.smithi146/config 2024-05-04T12:12:23.695 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:12:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:12:23.367+0000 7fe0daa0c640 -1 osd.2 0 log_to_monitors true 2024-05-04T12:12:24.194 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:24 smithi079 ceph-mon[27222]: from='osd.2 [v2:172.21.15.79:6818/3776007511,v1:172.21.15.79:6819/3776007511]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-04T12:12:24.276 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:24 smithi146 ceph-mon[35311]: from='osd.2 [v2:172.21.15.79:6818/3776007511,v1:172.21.15.79:6819/3776007511]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-04T12:12:25.240 INFO:teuthology.orchestra.run.smithi146.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-05-04T12:12:25.241 INFO:teuthology.orchestra.run.smithi146.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-05-04T12:12:25.241 INFO:teuthology.orchestra.run.smithi146.stderr: stderr: 10+0 records in 2024-05-04T12:12:25.241 INFO:teuthology.orchestra.run.smithi146.stderr:10+0 records out 2024-05-04T12:12:25.241 INFO:teuthology.orchestra.run.smithi146.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0115631 s, 907 MB/s 2024-05-04T12:12:25.241 INFO:teuthology.orchestra.run.smithi146.stderr:--> Zapping successful for: 2024-05-04T12:12:25.283 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:25 smithi146 ceph-mon[35311]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-04T12:12:25.283 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:25 smithi146 ceph-mon[35311]: from='osd.2 [v2:172.21.15.79:6818/3776007511,v1:172.21.15.79:6819/3776007511]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-05-04T12:12:25.283 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:25 smithi146 ceph-mon[35311]: osdmap e17: 3 total, 2 up, 3 in 2024-05-04T12:12:25.283 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:25 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-04T12:12:25.283 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:25 smithi146 ceph-mon[35311]: from='osd.2 [v2:172.21.15.79:6818/3776007511,v1:172.21.15.79:6819/3776007511]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2024-05-04T12:12:25.328 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:25 smithi079 ceph-mon[27222]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-04T12:12:25.328 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:25 smithi079 ceph-mon[27222]: from='osd.2 [v2:172.21.15.79:6818/3776007511,v1:172.21.15.79:6819/3776007511]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-05-04T12:12:25.328 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:25 smithi079 ceph-mon[27222]: osdmap e17: 3 total, 2 up, 3 in 2024-05-04T12:12:25.328 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:25 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-04T12:12:25.328 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:25 smithi079 ceph-mon[27222]: from='osd.2 [v2:172.21.15.79:6818/3776007511,v1:172.21.15.79:6819/3776007511]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2024-05-04T12:12:25.328 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:12:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:12:25.031+0000 7fe0cf86f700 -1 osd.2 0 waiting for initial osdmap 2024-05-04T12:12:25.328 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:12:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:12:25.038+0000 7fe0cbe65700 -1 osd.2 18 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-04T12:12:25.856 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d9b985c4-0a0e-11ef-bc97-c7b262605968 -- ceph orch daemon add osd smithi146:vg_nvme/lv_4 2024-05-04T12:12:26.101 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/d9b985c4-0a0e-11ef-bc97-c7b262605968/mon.smithi146/config 2024-05-04T12:12:26.219 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:26 smithi146 ceph-mon[35311]: from='osd.2 [v2:172.21.15.79:6818/3776007511,v1:172.21.15.79:6819/3776007511]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]': finished 2024-05-04T12:12:26.219 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:26 smithi146 ceph-mon[35311]: osdmap e18: 3 total, 2 up, 3 in 2024-05-04T12:12:26.219 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:26 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-04T12:12:26.219 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:26 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-04T12:12:26.407 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:26 smithi079 ceph-mon[27222]: from='osd.2 [v2:172.21.15.79:6818/3776007511,v1:172.21.15.79:6819/3776007511]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]': finished 2024-05-04T12:12:26.407 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:26 smithi079 ceph-mon[27222]: osdmap e18: 3 total, 2 up, 3 in 2024-05-04T12:12:26.407 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:26 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-04T12:12:26.407 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:26 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-04T12:12:27.310 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:27 smithi146 ceph-mon[35311]: purged_snaps scrub starts 2024-05-04T12:12:27.310 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:27 smithi146 ceph-mon[35311]: purged_snaps scrub ok 2024-05-04T12:12:27.311 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:27 smithi146 ceph-mon[35311]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-04T12:12:27.311 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:27 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-04T12:12:27.311 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:27 smithi146 ceph-mon[35311]: osd.2 [v2:172.21.15.79:6818/3776007511,v1:172.21.15.79:6819/3776007511] boot 2024-05-04T12:12:27.311 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:27 smithi146 ceph-mon[35311]: osdmap e19: 3 total, 3 up, 3 in 2024-05-04T12:12:27.311 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:27 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-04T12:12:27.311 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:27 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-04T12:12:27.311 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:27 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:12:27.311 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:27 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:12:27.371 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:27 smithi079 ceph-mon[27222]: purged_snaps scrub starts 2024-05-04T12:12:27.371 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:27 smithi079 ceph-mon[27222]: purged_snaps scrub ok 2024-05-04T12:12:27.371 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:27 smithi079 ceph-mon[27222]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-04T12:12:27.371 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:27 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-04T12:12:27.371 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:27 smithi079 ceph-mon[27222]: osd.2 [v2:172.21.15.79:6818/3776007511,v1:172.21.15.79:6819/3776007511] boot 2024-05-04T12:12:27.371 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:27 smithi079 ceph-mon[27222]: osdmap e19: 3 total, 3 up, 3 in 2024-05-04T12:12:27.371 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:27 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-04T12:12:27.371 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:27 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-04T12:12:27.371 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:27 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:12:27.371 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:27 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:12:27.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:12:27.664+0000 7f85c372d700 1 -- 172.21.15.146:0/906397971 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f85bc0ffc10 msgr2=0x7f85bc100030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:12:27.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:12:27.664+0000 7f85c372d700 1 --2- 172.21.15.146:0/906397971 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f85bc0ffc10 0x7f85bc100030 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f85b8009910 tx=0x7f85b8009c20 comp rx=0 tx=0).stop 2024-05-04T12:12:27.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:12:27.664+0000 7f85c372d700 1 -- 172.21.15.146:0/906397971 shutdown_connections 2024-05-04T12:12:27.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:12:27.664+0000 7f85c372d700 1 --2- 172.21.15.146:0/906397971 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f85bc100e10 0x7f85bc101270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:12:27.666 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:12:27.664+0000 7f85c372d700 1 --2- 172.21.15.146:0/906397971 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f85bc0ffc10 0x7f85bc100030 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:12:27.666 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:12:27.664+0000 7f85c372d700 1 -- 172.21.15.146:0/906397971 >> 172.21.15.146:0/906397971 conn(0x7f85bc0fb190 msgr2=0x7f85bc0fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:12:27.666 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:12:27.664+0000 7f85c372d700 1 -- 172.21.15.146:0/906397971 shutdown_connections 2024-05-04T12:12:27.666 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:12:27.664+0000 7f85c372d700 1 -- 172.21.15.146:0/906397971 wait complete. 2024-05-04T12:12:27.666 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:12:27.665+0000 7f85c372d700 1 Processor -- start 2024-05-04T12:12:27.666 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:12:27.665+0000 7f85c372d700 1 -- start start 2024-05-04T12:12:27.666 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:12:27.665+0000 7f85c372d700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f85bc0ffc10 0x7f85bc1964b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:12:27.666 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:12:27.665+0000 7f85c372d700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f85bc100e10 0x7f85bc1969f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:12:27.666 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:12:27.665+0000 7f85c372d700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85bc197010 con 0x7f85bc0ffc10 2024-05-04T12:12:27.666 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:12:27.665+0000 7f85c372d700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85bc197150 con 0x7f85bc100e10 2024-05-04T12:12:27.666 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:12:27.665+0000 7f85c0cc8700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f85bc100e10 0x7f85bc1969f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:12:27.666 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:12:27.665+0000 7f85c14c9700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f85bc0ffc10 0x7f85bc1964b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:12:27.666 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:12:27.665+0000 7f85c0cc8700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f85bc100e10 0x7f85bc1969f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:51818/0 (socket says 172.21.15.146:51818) 2024-05-04T12:12:27.666 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:12:27.665+0000 7f85c0cc8700 1 -- 172.21.15.146:0/721350231 learned_addr learned my addr 172.21.15.146:0/721350231 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-05-04T12:12:27.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:12:27.665+0000 7f85c0cc8700 1 -- 172.21.15.146:0/721350231 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f85bc0ffc10 msgr2=0x7f85bc1964b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:12:27.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:12:27.665+0000 7f85c0cc8700 1 --2- 172.21.15.146:0/721350231 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f85bc0ffc10 0x7f85bc1964b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:12:27.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:12:27.665+0000 7f85c0cc8700 1 -- 172.21.15.146:0/721350231 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85b80095f0 con 0x7f85bc100e10 2024-05-04T12:12:27.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:12:27.666+0000 7f85c0cc8700 1 --2- 172.21.15.146:0/721350231 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f85bc100e10 0x7f85bc1969f0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f85ac00c8a0 tx=0x7f85ac00cbb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:12:27.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:12:27.666+0000 7f85b27fc700 1 -- 172.21.15.146:0/721350231 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f85ac0078e0 con 0x7f85bc100e10 2024-05-04T12:12:27.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:12:27.666+0000 7f85b27fc700 1 -- 172.21.15.146:0/721350231 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f85ac00f450 con 0x7f85bc100e10 2024-05-04T12:12:27.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:12:27.666+0000 7f85c372d700 1 -- 172.21.15.146:0/721350231 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85bc19bc00 con 0x7f85bc100e10 2024-05-04T12:12:27.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:12:27.666+0000 7f85b27fc700 1 -- 172.21.15.146:0/721350231 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f85ac00e5e0 con 0x7f85bc100e10 2024-05-04T12:12:27.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:12:27.666+0000 7f85c372d700 1 -- 172.21.15.146:0/721350231 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85bc19c150 con 0x7f85bc100e10 2024-05-04T12:12:27.668 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:12:27.667+0000 7f85c372d700 1 -- 172.21.15.146:0/721350231 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85bc108700 con 0x7f85bc100e10 2024-05-04T12:12:27.668 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:12:27.667+0000 7f85b27fc700 1 -- 172.21.15.146:0/721350231 <== mon.1 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 18) v1 ==== 90256+0+0 (secure 0 0 0) 0x7f85ac00f5c0 con 0x7f85bc100e10 2024-05-04T12:12:27.668 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:12:27.667+0000 7f85b27fc700 1 --2- 172.21.15.146:0/721350231 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f85a806c460 0x7f85a806e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:12:27.668 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:12:27.668+0000 7f85b27fc700 1 -- 172.21.15.146:0/721350231 <== mon.1 v2:172.21.15.146:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 2846+0+0 (secure 0 0 0) 0x7f85ac08ac50 con 0x7f85bc100e10 2024-05-04T12:12:27.668 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:12:27.668+0000 7f85c14c9700 1 --2- 172.21.15.146:0/721350231 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f85a806c460 0x7f85a806e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:12:27.668 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:12:27.668+0000 7f85c14c9700 1 --2- 172.21.15.146:0/721350231 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f85a806c460 0x7f85a806e920 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f85b8006010 tx=0x7f85b800b540 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:12:27.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:12:27.672+0000 7f85b27fc700 1 -- 172.21.15.146:0/721350231 <== mon.1 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f85ac090020 con 0x7f85bc100e10 2024-05-04T12:12:27.870 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:12:27.871+0000 7f85c372d700 1 -- 172.21.15.146:0/721350231 --> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi146:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f85bc068c90 con 0x7f85a806c460 2024-05-04T12:12:28.276 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:28 smithi146 ceph-mon[35311]: osdmap e20: 3 total, 3 up, 3 in 2024-05-04T12:12:28.276 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:28 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-05-04T12:12:28.276 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:28 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-04T12:12:28.277 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:28 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-04T12:12:28.277 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:28 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-04T12:12:28.340 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:28 smithi079 ceph-mon[27222]: osdmap e20: 3 total, 3 up, 3 in 2024-05-04T12:12:28.340 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:28 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-05-04T12:12:28.340 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:28 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-04T12:12:28.341 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:28 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-04T12:12:28.341 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:28 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-04T12:12:29.325 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:29 smithi146 ceph-mon[35311]: pgmap v76: 0 pgs: ; 0 B data, 79 MiB used, 268 GiB / 268 GiB avail 2024-05-04T12:12:29.326 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:29 smithi146 ceph-mon[35311]: from='client.24129 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi146:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-04T12:12:29.326 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:29 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-05-04T12:12:29.326 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:29 smithi146 ceph-mon[35311]: osdmap e21: 3 total, 3 up, 3 in 2024-05-04T12:12:29.326 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:29 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-05-04T12:12:29.413 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:29 smithi079 ceph-mon[27222]: pgmap v76: 0 pgs: ; 0 B data, 79 MiB used, 268 GiB / 268 GiB avail 2024-05-04T12:12:29.413 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:29 smithi079 ceph-mon[27222]: from='client.24129 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi146:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-04T12:12:29.414 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:29 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-05-04T12:12:29.414 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:29 smithi079 ceph-mon[27222]: osdmap e21: 3 total, 3 up, 3 in 2024-05-04T12:12:29.414 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:29 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-05-04T12:12:30.313 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:30 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-05-04T12:12:30.313 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:30 smithi079 ceph-mon[27222]: osdmap e22: 3 total, 3 up, 3 in 2024-05-04T12:12:30.313 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:30 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:12:30.313 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:30 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:12:30.314 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:30 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-04T12:12:30.314 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:30 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-04T12:12:30.314 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:30 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-04T12:12:30.314 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:30 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:12:30.314 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:30 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-04T12:12:30.314 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:30 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-04T12:12:30.314 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:30 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:12:30.314 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:30 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-04T12:12:30.526 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:30 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-05-04T12:12:30.526 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:30 smithi146 ceph-mon[35311]: osdmap e22: 3 total, 3 up, 3 in 2024-05-04T12:12:30.526 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:30 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:12:30.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:30 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:12:30.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:30 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-04T12:12:30.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:30 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-04T12:12:30.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:30 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-04T12:12:30.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:30 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:12:30.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:30 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-04T12:12:30.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:30 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-04T12:12:30.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:30 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:12:30.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:30 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-04T12:12:30.586 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:12:30 smithi079 sudo[65705]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-04T12:12:30.586 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:12:30 smithi079 sudo[65705]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-04T12:12:30.586 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:12:30 smithi079 sudo[65705]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-04T12:12:30.586 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:12:30 smithi079 sudo[65705]: pam_unix(sudo:session): session closed for user root 2024-05-04T12:12:30.861 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:12:30 smithi079 sudo[65741]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-04T12:12:30.861 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:12:30 smithi079 sudo[65741]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-04T12:12:30.861 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:12:30 smithi079 sudo[65741]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-04T12:12:30.861 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:12:30 smithi079 sudo[65741]: pam_unix(sudo:session): session closed for user root 2024-05-04T12:12:31.123 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:12:30 smithi079 sudo[65761]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-04T12:12:31.124 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:12:30 smithi079 sudo[65761]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-04T12:12:31.124 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:12:30 smithi079 sudo[65761]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-04T12:12:31.124 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:12:30 smithi079 sudo[65761]: pam_unix(sudo:session): session closed for user root 2024-05-04T12:12:31.124 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:31 smithi079 ceph-mon[27222]: pgmap v79: 1 pgs: 1 unknown; 0 B data, 79 MiB used, 268 GiB / 268 GiB avail 2024-05-04T12:12:31.124 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:31 smithi079 ceph-mon[27222]: Detected new or changed devices on smithi079 2024-05-04T12:12:31.124 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:31 smithi079 ceph-mon[27222]: Adjusting osd_memory_target on smithi079 to 3946M 2024-05-04T12:12:31.124 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:31 smithi079 ceph-mon[27222]: from='client.? 172.21.15.146:0/1823951128' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "67d66fd8-d092-4ba0-8418-799689ae0a04"}]: dispatch 2024-05-04T12:12:31.124 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:31 smithi079 ceph-mon[27222]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "67d66fd8-d092-4ba0-8418-799689ae0a04"}]: dispatch 2024-05-04T12:12:31.124 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:31 smithi079 ceph-mon[27222]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "67d66fd8-d092-4ba0-8418-799689ae0a04"}]': finished 2024-05-04T12:12:31.125 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:31 smithi079 ceph-mon[27222]: osdmap e23: 4 total, 3 up, 4 in 2024-05-04T12:12:31.125 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:31 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-04T12:12:31.125 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:31 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:12:31.125 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:31 smithi079 ceph-mon[27222]: from='client.? 172.21.15.146:0/608479658' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-04T12:12:31.392 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:12:31 smithi079 sudo[65768]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-04T12:12:31.392 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:12:31 smithi079 sudo[65768]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-04T12:12:31.393 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:12:31 smithi079 sudo[65768]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-04T12:12:31.393 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:12:31 smithi079 sudo[65768]: pam_unix(sudo:session): session closed for user root 2024-05-04T12:12:31.526 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:31 smithi146 ceph-mon[35311]: pgmap v79: 1 pgs: 1 unknown; 0 B data, 79 MiB used, 268 GiB / 268 GiB avail 2024-05-04T12:12:31.526 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:31 smithi146 ceph-mon[35311]: Detected new or changed devices on smithi079 2024-05-04T12:12:31.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:31 smithi146 ceph-mon[35311]: Adjusting osd_memory_target on smithi079 to 3946M 2024-05-04T12:12:31.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:31 smithi146 ceph-mon[35311]: from='client.? 172.21.15.146:0/1823951128' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "67d66fd8-d092-4ba0-8418-799689ae0a04"}]: dispatch 2024-05-04T12:12:31.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:31 smithi146 ceph-mon[35311]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "67d66fd8-d092-4ba0-8418-799689ae0a04"}]: dispatch 2024-05-04T12:12:31.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:31 smithi146 ceph-mon[35311]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "67d66fd8-d092-4ba0-8418-799689ae0a04"}]': finished 2024-05-04T12:12:31.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:31 smithi146 ceph-mon[35311]: osdmap e23: 4 total, 3 up, 4 in 2024-05-04T12:12:31.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:31 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-04T12:12:31.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:31 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:12:31.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:31 smithi146 ceph-mon[35311]: from='client.? 172.21.15.146:0/608479658' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-04T12:12:31.658 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:12:31 smithi079 sudo[65869]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-04T12:12:31.658 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:12:31 smithi079 sudo[65869]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-04T12:12:31.658 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:12:31 smithi079 sudo[65869]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-04T12:12:31.659 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:12:31 smithi079 sudo[65869]: pam_unix(sudo:session): session closed for user root 2024-05-04T12:12:31.911 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:12:31 smithi079 sudo[65901]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-04T12:12:31.911 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:12:31 smithi079 sudo[65901]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-04T12:12:31.911 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:12:31 smithi079 sudo[65901]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-04T12:12:31.911 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:12:31 smithi079 sudo[65901]: pam_unix(sudo:session): session closed for user root 2024-05-04T12:12:32.191 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:31 smithi079 sudo[65906]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-04T12:12:32.192 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:31 smithi079 sudo[65906]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-04T12:12:32.192 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:31 smithi079 sudo[65906]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-04T12:12:32.192 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:31 smithi079 sudo[65906]: pam_unix(sudo:session): session closed for user root 2024-05-04T12:12:32.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:32 smithi079 sudo[65956]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-04T12:12:32.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:32 smithi079 sudo[65956]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-04T12:12:32.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:32 smithi079 sudo[65956]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-04T12:12:32.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:32 smithi079 sudo[65956]: pam_unix(sudo:session): session closed for user root 2024-05-04T12:12:32.696 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:32 smithi146 sudo[39606]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-04T12:12:32.696 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:32 smithi146 sudo[39606]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-04T12:12:32.696 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:32 smithi146 sudo[39606]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-04T12:12:32.696 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:32 smithi146 sudo[39606]: pam_unix(sudo:session): session closed for user root 2024-05-04T12:12:32.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:12:32.704+0000 7f85b27fc700 1 -- 172.21.15.146:0/721350231 <== mon.1 v2:172.21.15.146:3300/0 7 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f85ac056320 con 0x7f85bc100e10 2024-05-04T12:12:32.944 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:32 smithi079 ceph-mon[27222]: pgmap v81: 1 pgs: 1 unknown; 0 B data, 79 MiB used, 268 GiB / 268 GiB avail 2024-05-04T12:12:32.945 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:32 smithi079 ceph-mon[27222]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-04T12:12:32.945 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:32 smithi079 ceph-mon[27222]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-04T12:12:32.945 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:32 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "mon metadata", "id": "smithi079"}]: dispatch 2024-05-04T12:12:32.945 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:32 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "mon metadata", "id": "smithi146"}]: dispatch 2024-05-04T12:12:32.945 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:32 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "mon metadata", "id": "smithi079"}]: dispatch 2024-05-04T12:12:32.945 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:32 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "mon metadata", "id": "smithi146"}]: dispatch 2024-05-04T12:12:33.026 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:32 smithi146 sudo[39795]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-04T12:12:33.026 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:32 smithi146 sudo[39795]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-04T12:12:33.027 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:32 smithi146 sudo[39795]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-04T12:12:33.027 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:32 smithi146 sudo[39795]: pam_unix(sudo:session): session closed for user root 2024-05-04T12:12:33.027 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:32 smithi146 ceph-mon[35311]: pgmap v81: 1 pgs: 1 unknown; 0 B data, 79 MiB used, 268 GiB / 268 GiB avail 2024-05-04T12:12:33.027 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:32 smithi146 ceph-mon[35311]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-04T12:12:33.027 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:32 smithi146 ceph-mon[35311]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-04T12:12:33.027 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:32 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "mon metadata", "id": "smithi079"}]: dispatch 2024-05-04T12:12:33.027 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:32 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "mon metadata", "id": "smithi146"}]: dispatch 2024-05-04T12:12:33.027 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:32 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "mon metadata", "id": "smithi079"}]: dispatch 2024-05-04T12:12:33.027 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:32 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "mon metadata", "id": "smithi146"}]: dispatch 2024-05-04T12:12:33.945 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:33 smithi079 ceph-mon[27222]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-04T12:12:33.945 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:33 smithi079 ceph-mon[27222]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-04T12:12:33.945 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:33 smithi079 ceph-mon[27222]: mgrmap e19: smithi079.dgcpwl(active, since 2m), standbys: smithi146.eubqih 2024-05-04T12:12:33.945 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:33 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:12:34.026 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:33 smithi146 ceph-mon[35311]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-04T12:12:34.026 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:33 smithi146 ceph-mon[35311]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-04T12:12:34.026 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:33 smithi146 ceph-mon[35311]: mgrmap e19: smithi079.dgcpwl(active, since 2m), standbys: smithi146.eubqih 2024-05-04T12:12:34.026 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:33 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:12:34.959 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:34 smithi146 ceph-mon[35311]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-05-04T12:12:34.959 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:34 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:12:34.959 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:34 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:12:35.195 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:34 smithi079 ceph-mon[27222]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-05-04T12:12:35.195 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:34 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:12:35.195 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:34 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:12:36.026 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:35 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-04T12:12:36.027 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:35 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-04T12:12:36.027 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:35 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:12:36.195 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:35 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-04T12:12:36.195 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:35 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-04T12:12:36.195 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:35 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:12:37.240 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:37 smithi146 ceph-mon[35311]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-05-04T12:12:37.444 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:37 smithi079 ceph-mon[27222]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-05-04T12:12:38.276 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:38 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-05-04T12:12:38.276 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:38 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-04T12:12:38.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:38 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-05-04T12:12:38.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:38 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-04T12:12:39.276 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:39 smithi146 ceph-mon[35311]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 4.5 KiB/s rd, 196 KiB/s wr, 13 op/s 2024-05-04T12:12:39.276 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:39 smithi146 ceph-mon[35311]: Deploying daemon osd.3 on smithi146 2024-05-04T12:12:39.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:39 smithi079 ceph-mon[27222]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 4.5 KiB/s rd, 196 KiB/s wr, 13 op/s 2024-05-04T12:12:39.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:39 smithi079 ceph-mon[27222]: Deploying daemon osd.3 on smithi146 2024-05-04T12:12:41.276 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:41 smithi146 ceph-mon[35311]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-05-04T12:12:41.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:41 smithi079 ceph-mon[27222]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-05-04T12:12:42.430 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:42 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-04T12:12:42.444 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:42 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-04T12:12:43.304 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:43 smithi146 ceph-mon[35311]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-05-04T12:12:43.444 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:43 smithi079 ceph-mon[27222]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-05-04T12:12:44.695 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:44 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:12:44.695 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:44 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:12:44.695 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:44 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-04T12:12:44.695 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:44 smithi079 ceph-mon[27222]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-05-04T12:12:44.776 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:44 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:12:44.776 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:44 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:12:44.777 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:44 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-04T12:12:44.777 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:44 smithi146 ceph-mon[35311]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-05-04T12:12:46.954 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:12:46.955+0000 7f85b27fc700 1 -- 172.21.15.146:0/721350231 <== mgr.14219 v2:172.21.15.79:6800/402764217 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f85bc068c90 con 0x7f85a806c460 2024-05-04T12:12:46.955 INFO:teuthology.orchestra.run.smithi146.stdout:Created osd(s) 3 on host 'smithi146' 2024-05-04T12:12:46.956 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:12:46.957+0000 7f85c372d700 1 -- 172.21.15.146:0/721350231 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f85a806c460 msgr2=0x7f85a806e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:12:46.957 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:12:46.957+0000 7f85c372d700 1 --2- 172.21.15.146:0/721350231 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f85a806c460 0x7f85a806e920 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f85b8006010 tx=0x7f85b800b540 comp rx=0 tx=0).stop 2024-05-04T12:12:46.957 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:12:46.957+0000 7f85c372d700 1 -- 172.21.15.146:0/721350231 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f85bc100e10 msgr2=0x7f85bc1969f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:12:46.957 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:12:46.957+0000 7f85c372d700 1 --2- 172.21.15.146:0/721350231 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f85bc100e10 0x7f85bc1969f0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f85ac00c8a0 tx=0x7f85ac00cbb0 comp rx=0 tx=0).stop 2024-05-04T12:12:46.957 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:12:46.957+0000 7f85c372d700 1 -- 172.21.15.146:0/721350231 shutdown_connections 2024-05-04T12:12:46.957 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:12:46.957+0000 7f85c372d700 1 --2- 172.21.15.146:0/721350231 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f85a806c460 0x7f85a806e920 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:12:46.957 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:12:46.957+0000 7f85c372d700 1 --2- 172.21.15.146:0/721350231 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f85bc0ffc10 0x7f85bc1964b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:12:46.957 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:12:46.957+0000 7f85c372d700 1 --2- 172.21.15.146:0/721350231 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f85bc100e10 0x7f85bc1969f0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:12:46.957 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:12:46.957+0000 7f85c372d700 1 -- 172.21.15.146:0/721350231 >> 172.21.15.146:0/721350231 conn(0x7f85bc0fb190 msgr2=0x7f85bc068580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:12:46.957 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:12:46.957+0000 7f85c372d700 1 -- 172.21.15.146:0/721350231 shutdown_connections 2024-05-04T12:12:46.957 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:12:46.957+0000 7f85c372d700 1 -- 172.21.15.146:0/721350231 wait complete. 2024-05-04T12:12:47.256 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:47 smithi146 ceph-mon[35311]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-05-04T12:12:47.256 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:47 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:12:47.256 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:47 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:12:47.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:47 smithi079 ceph-mon[27222]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-05-04T12:12:47.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:47 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:12:47.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:47 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:12:47.561 DEBUG:teuthology.orchestra.run.smithi146:osd.3> sudo journalctl -f -n 0 -u ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968@osd.3.service 2024-05-04T12:12:47.564 INFO:tasks.cephadm:Deploying osd.4 on smithi146 with /dev/vg_nvme/lv_3... 2024-05-04T12:12:47.564 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d9b985c4-0a0e-11ef-bc97-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-05-04T12:12:47.937 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/d9b985c4-0a0e-11ef-bc97-c7b262605968/mon.smithi146/config 2024-05-04T12:12:48.286 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:48 smithi146 ceph-mon[35311]: from='osd.3 [v2:172.21.15.146:6800/1187278645,v1:172.21.15.146:6801/1187278645]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-05-04T12:12:48.286 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:48 smithi146 ceph-mon[35311]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-05-04T12:12:48.444 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:48 smithi079 ceph-mon[27222]: from='osd.3 [v2:172.21.15.146:6800/1187278645,v1:172.21.15.146:6801/1187278645]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-05-04T12:12:48.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:48 smithi079 ceph-mon[27222]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-05-04T12:12:49.311 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:49 smithi146 ceph-mon[35311]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-05-04T12:12:49.312 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:49 smithi146 ceph-mon[35311]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-05-04T12:12:49.312 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:49 smithi146 ceph-mon[35311]: osdmap e24: 4 total, 3 up, 4 in 2024-05-04T12:12:49.312 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:49 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-04T12:12:49.312 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:49 smithi146 ceph-mon[35311]: from='osd.3 [v2:172.21.15.146:6800/1187278645,v1:172.21.15.146:6801/1187278645]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-05-04T12:12:49.312 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:49 smithi146 ceph-mon[35311]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-05-04T12:12:49.312 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:49 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:12:49.312 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:49 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:12:49.312 INFO:journalctl@ceph.osd.3.smithi146.stdout:May 04 12:12:49 smithi146 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-3[41233]: 2024-05-04T12:12:49.042+0000 7fc1fe176700 -1 osd.3 0 waiting for initial osdmap 2024-05-04T12:12:49.312 INFO:journalctl@ceph.osd.3.smithi146.stdout:May 04 12:12:49 smithi146 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-3[41233]: 2024-05-04T12:12:49.048+0000 7fc1f6f65700 -1 osd.3 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-04T12:12:49.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:49 smithi079 ceph-mon[27222]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-05-04T12:12:49.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:49 smithi079 ceph-mon[27222]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-05-04T12:12:49.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:49 smithi079 ceph-mon[27222]: osdmap e24: 4 total, 3 up, 4 in 2024-05-04T12:12:49.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:49 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-04T12:12:49.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:49 smithi079 ceph-mon[27222]: from='osd.3 [v2:172.21.15.146:6800/1187278645,v1:172.21.15.146:6801/1187278645]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-05-04T12:12:49.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:49 smithi079 ceph-mon[27222]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-05-04T12:12:49.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:49 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:12:49.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:49 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:12:50.005 INFO:teuthology.orchestra.run.smithi146.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-05-04T12:12:50.005 INFO:teuthology.orchestra.run.smithi146.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-05-04T12:12:50.005 INFO:teuthology.orchestra.run.smithi146.stderr: stderr: 10+0 records in 2024-05-04T12:12:50.005 INFO:teuthology.orchestra.run.smithi146.stderr:10+0 records out 2024-05-04T12:12:50.006 INFO:teuthology.orchestra.run.smithi146.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.012454 s, 842 MB/s 2024-05-04T12:12:50.006 INFO:teuthology.orchestra.run.smithi146.stderr:--> Zapping successful for: 2024-05-04T12:12:50.194 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:50 smithi079 ceph-mon[27222]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]': finished 2024-05-04T12:12:50.195 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:50 smithi079 ceph-mon[27222]: osdmap e25: 4 total, 3 up, 4 in 2024-05-04T12:12:50.195 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:50 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-04T12:12:50.195 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:50 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-04T12:12:50.276 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:50 smithi146 ceph-mon[35311]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]': finished 2024-05-04T12:12:50.276 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:50 smithi146 ceph-mon[35311]: osdmap e25: 4 total, 3 up, 4 in 2024-05-04T12:12:50.276 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:50 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-04T12:12:50.277 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:50 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-04T12:12:51.080 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:51 smithi146 ceph-mon[35311]: purged_snaps scrub starts 2024-05-04T12:12:51.080 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:51 smithi146 ceph-mon[35311]: purged_snaps scrub ok 2024-05-04T12:12:51.080 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:51 smithi146 ceph-mon[35311]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-05-04T12:12:51.080 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:51 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-04T12:12:51.081 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:51 smithi146 ceph-mon[35311]: osd.3 [v2:172.21.15.146:6800/1187278645,v1:172.21.15.146:6801/1187278645] boot 2024-05-04T12:12:51.081 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:51 smithi146 ceph-mon[35311]: osdmap e26: 4 total, 4 up, 4 in 2024-05-04T12:12:51.081 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:51 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-04T12:12:51.081 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:51 smithi146 ceph-mon[35311]: osdmap e27: 4 total, 4 up, 4 in 2024-05-04T12:12:51.103 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d9b985c4-0a0e-11ef-bc97-c7b262605968 -- ceph orch daemon add osd smithi146:vg_nvme/lv_3 2024-05-04T12:12:51.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:51 smithi079 ceph-mon[27222]: purged_snaps scrub starts 2024-05-04T12:12:51.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:51 smithi079 ceph-mon[27222]: purged_snaps scrub ok 2024-05-04T12:12:51.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:51 smithi079 ceph-mon[27222]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-05-04T12:12:51.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:51 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-04T12:12:51.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:51 smithi079 ceph-mon[27222]: osd.3 [v2:172.21.15.146:6800/1187278645,v1:172.21.15.146:6801/1187278645] boot 2024-05-04T12:12:51.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:51 smithi079 ceph-mon[27222]: osdmap e26: 4 total, 4 up, 4 in 2024-05-04T12:12:51.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:51 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-04T12:12:51.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:51 smithi079 ceph-mon[27222]: osdmap e27: 4 total, 4 up, 4 in 2024-05-04T12:12:51.734 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/d9b985c4-0a0e-11ef-bc97-c7b262605968/mon.smithi146/config 2024-05-04T12:12:53.071 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:53 smithi146 ceph-mon[35311]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-05-04T12:12:53.072 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:53 smithi146 ceph-mon[35311]: osdmap e28: 4 total, 4 up, 4 in 2024-05-04T12:12:53.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:53 smithi079 ceph-mon[27222]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-05-04T12:12:53.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:53 smithi079 ceph-mon[27222]: osdmap e28: 4 total, 4 up, 4 in 2024-05-04T12:12:53.953 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:12:53.935+0000 7f0722ce1700 1 -- 172.21.15.146:0/4200902521 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f071c1003e0 msgr2=0x7f071c1027d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:12:53.953 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:12:53.935+0000 7f0722ce1700 1 --2- 172.21.15.146:0/4200902521 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f071c1003e0 0x7f071c1027d0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f0710009910 tx=0x7f0710009c20 comp rx=0 tx=0).stop 2024-05-04T12:12:53.953 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:12:53.935+0000 7f0722ce1700 1 -- 172.21.15.146:0/4200902521 shutdown_connections 2024-05-04T12:12:53.953 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:12:53.935+0000 7f0722ce1700 1 --2- 172.21.15.146:0/4200902521 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f071c102da0 0x7f071c105190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:12:53.953 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:12:53.935+0000 7f0722ce1700 1 --2- 172.21.15.146:0/4200902521 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f071c1003e0 0x7f071c1027d0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:12:53.954 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:12:53.935+0000 7f0722ce1700 1 -- 172.21.15.146:0/4200902521 >> 172.21.15.146:0/4200902521 conn(0x7f071c0fa490 msgr2=0x7f071c0fc8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:12:53.954 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:12:53.935+0000 7f0722ce1700 1 -- 172.21.15.146:0/4200902521 shutdown_connections 2024-05-04T12:12:53.954 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:12:53.935+0000 7f0722ce1700 1 -- 172.21.15.146:0/4200902521 wait complete. 2024-05-04T12:12:53.954 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:12:53.936+0000 7f0722ce1700 1 Processor -- start 2024-05-04T12:12:53.954 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:12:53.936+0000 7f0722ce1700 1 -- start start 2024-05-04T12:12:53.954 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:12:53.936+0000 7f0722ce1700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f071c1003e0 0x7f071c198770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:12:53.954 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:12:53.936+0000 7f0722ce1700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f071c102da0 0x7f071c198cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:12:53.954 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:12:53.936+0000 7f0722ce1700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f071c1992d0 con 0x7f071c1003e0 2024-05-04T12:12:53.954 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:12:53.936+0000 7f0722ce1700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f071c199410 con 0x7f071c102da0 2024-05-04T12:12:53.954 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:12:53.936+0000 7f071bfff700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f071c102da0 0x7f071c198cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:12:53.954 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:12:53.936+0000 7f071bfff700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f071c102da0 0x7f071c198cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:34050/0 (socket says 172.21.15.146:34050) 2024-05-04T12:12:53.954 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:12:53.936+0000 7f071bfff700 1 -- 172.21.15.146:0/1338561717 learned_addr learned my addr 172.21.15.146:0/1338561717 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-05-04T12:12:53.954 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:12:53.936+0000 7f0720a7d700 1 --2- 172.21.15.146:0/1338561717 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f071c1003e0 0x7f071c198770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:12:53.955 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:12:53.937+0000 7f071bfff700 1 -- 172.21.15.146:0/1338561717 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f071c1003e0 msgr2=0x7f071c198770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:12:53.955 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:12:53.937+0000 7f071bfff700 1 --2- 172.21.15.146:0/1338561717 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f071c1003e0 0x7f071c198770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:12:53.955 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:12:53.937+0000 7f071bfff700 1 -- 172.21.15.146:0/1338561717 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07100095f0 con 0x7f071c102da0 2024-05-04T12:12:53.955 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:12:53.937+0000 7f071bfff700 1 --2- 172.21.15.146:0/1338561717 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f071c102da0 0x7f071c198cb0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f070800b4b0 tx=0x7f070800b7c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:12:53.955 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:12:53.937+0000 7f0719ffb700 1 -- 172.21.15.146:0/1338561717 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0708011800 con 0x7f071c102da0 2024-05-04T12:12:53.955 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:12:53.937+0000 7f0719ffb700 1 -- 172.21.15.146:0/1338561717 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f0708011e10 con 0x7f071c102da0 2024-05-04T12:12:53.955 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:12:53.937+0000 7f0722ce1700 1 -- 172.21.15.146:0/1338561717 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f071c19dec0 con 0x7f071c102da0 2024-05-04T12:12:53.955 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:12:53.937+0000 7f0719ffb700 1 -- 172.21.15.146:0/1338561717 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f070800f2e0 con 0x7f071c102da0 2024-05-04T12:12:53.955 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:12:53.937+0000 7f0722ce1700 1 -- 172.21.15.146:0/1338561717 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f071c19e3e0 con 0x7f071c102da0 2024-05-04T12:12:53.955 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:12:53.938+0000 7f0722ce1700 1 -- 172.21.15.146:0/1338561717 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f071c1087a0 con 0x7f071c102da0 2024-05-04T12:12:53.955 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:12:53.939+0000 7f0719ffb700 1 -- 172.21.15.146:0/1338561717 <== mon.1 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f0708010380 con 0x7f071c102da0 2024-05-04T12:12:53.955 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:12:53.939+0000 7f0719ffb700 1 --2- 172.21.15.146:0/1338561717 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f070c06c4e0 0x7f070c06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:12:53.955 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:12:53.939+0000 7f0719ffb700 1 -- 172.21.15.146:0/1338561717 <== mon.1 v2:172.21.15.146:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 3774+0+0 (secure 0 0 0) 0x7f070808ad20 con 0x7f071c102da0 2024-05-04T12:12:53.955 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:12:53.939+0000 7f0720a7d700 1 --2- 172.21.15.146:0/1338561717 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f070c06c4e0 0x7f070c06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:12:53.956 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:12:53.940+0000 7f0720a7d700 1 --2- 172.21.15.146:0/1338561717 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f070c06c4e0 0x7f070c06e9a0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f0710005090 tx=0x7f071000b540 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:12:53.956 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:12:53.944+0000 7f0719ffb700 1 -- 172.21.15.146:0/1338561717 <== mon.1 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f0708059630 con 0x7f071c102da0 2024-05-04T12:12:54.142 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:12:54.144+0000 7f0722ce1700 1 -- 172.21.15.146:0/1338561717 --> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi146:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f071c19ea80 con 0x7f070c06c4e0 2024-05-04T12:12:54.276 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:54 smithi146 ceph-mon[35311]: osdmap e29: 4 total, 4 up, 4 in 2024-05-04T12:12:54.277 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:54 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:12:54.277 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:54 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:12:54.277 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:54 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-04T12:12:54.277 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:54 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:12:54.277 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:54 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-04T12:12:54.277 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:54 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-04T12:12:54.277 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:54 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:12:54.277 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:54 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-04T12:12:54.277 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:54 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-04T12:12:54.277 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:54 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-04T12:12:54.277 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:54 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:12:54.277 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:54 smithi146 ceph-mon[35311]: osdmap e30: 4 total, 4 up, 4 in 2024-05-04T12:12:54.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:54 smithi079 ceph-mon[27222]: osdmap e29: 4 total, 4 up, 4 in 2024-05-04T12:12:54.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:54 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:12:54.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:54 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:12:54.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:54 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-04T12:12:54.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:54 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:12:54.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:54 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-04T12:12:54.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:54 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-04T12:12:54.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:54 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:12:54.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:54 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-04T12:12:54.446 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:54 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-04T12:12:54.446 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:54 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-04T12:12:54.446 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:54 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:12:54.446 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:54 smithi079 ceph-mon[27222]: osdmap e30: 4 total, 4 up, 4 in 2024-05-04T12:12:55.265 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:55 smithi146 ceph-mon[35311]: pgmap v98: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-05-04T12:12:55.266 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:55 smithi146 ceph-mon[35311]: Detected new or changed devices on smithi146 2024-05-04T12:12:55.266 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:55 smithi146 ceph-mon[35311]: Adjusting osd_memory_target on smithi146 to 14912M 2024-05-04T12:12:55.266 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:55 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-04T12:12:55.266 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:55 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-04T12:12:55.266 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:55 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-04T12:12:55.444 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:55 smithi079 ceph-mon[27222]: pgmap v98: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-05-04T12:12:55.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:55 smithi079 ceph-mon[27222]: Detected new or changed devices on smithi146 2024-05-04T12:12:55.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:55 smithi079 ceph-mon[27222]: Adjusting osd_memory_target on smithi146 to 14912M 2024-05-04T12:12:55.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:55 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-04T12:12:55.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:55 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-04T12:12:55.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:55 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-04T12:12:56.276 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:56 smithi146 ceph-mon[35311]: from='client.24151 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi146:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-04T12:12:56.444 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:56 smithi079 ceph-mon[27222]: from='client.24151 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi146:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-04T12:12:57.195 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:57 smithi079 ceph-mon[27222]: pgmap v100: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 126 KiB/s, 0 objects/s recovering 2024-05-04T12:12:57.195 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:57 smithi079 ceph-mon[27222]: from='client.? 172.21.15.146:0/1953313436' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "845a6602-d767-45c8-9fd1-1cbbe89adb23"}]: dispatch 2024-05-04T12:12:57.195 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:57 smithi079 ceph-mon[27222]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "845a6602-d767-45c8-9fd1-1cbbe89adb23"}]: dispatch 2024-05-04T12:12:57.195 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:57 smithi079 ceph-mon[27222]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "845a6602-d767-45c8-9fd1-1cbbe89adb23"}]': finished 2024-05-04T12:12:57.195 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:57 smithi079 ceph-mon[27222]: osdmap e31: 5 total, 4 up, 5 in 2024-05-04T12:12:57.195 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:57 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-04T12:12:57.195 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:57 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-04T12:12:57.276 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:57 smithi146 ceph-mon[35311]: pgmap v100: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 126 KiB/s, 0 objects/s recovering 2024-05-04T12:12:57.277 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:57 smithi146 ceph-mon[35311]: from='client.? 172.21.15.146:0/1953313436' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "845a6602-d767-45c8-9fd1-1cbbe89adb23"}]: dispatch 2024-05-04T12:12:57.277 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:57 smithi146 ceph-mon[35311]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "845a6602-d767-45c8-9fd1-1cbbe89adb23"}]: dispatch 2024-05-04T12:12:57.277 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:57 smithi146 ceph-mon[35311]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "845a6602-d767-45c8-9fd1-1cbbe89adb23"}]': finished 2024-05-04T12:12:57.277 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:57 smithi146 ceph-mon[35311]: osdmap e31: 5 total, 4 up, 5 in 2024-05-04T12:12:57.277 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:57 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-04T12:12:57.277 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:57 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-04T12:12:58.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:58 smithi079 ceph-mon[27222]: from='client.? 172.21.15.146:0/366526155' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-04T12:12:58.477 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:58 smithi146 ceph-mon[35311]: from='client.? 172.21.15.146:0/366526155' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-04T12:12:59.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:12:59 smithi079 ceph-mon[27222]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 103 KiB/s, 0 objects/s recovering 2024-05-04T12:12:59.526 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:12:59 smithi146 ceph-mon[35311]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 103 KiB/s, 0 objects/s recovering 2024-05-04T12:13:01.210 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:01 smithi146 ceph-mon[35311]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-05-04T12:13:01.365 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:01 smithi079 ceph-mon[27222]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-05-04T12:13:03.277 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:03 smithi146 ceph-mon[35311]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-05-04T12:13:03.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:03 smithi079 ceph-mon[27222]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-05-04T12:13:04.276 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:04 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-05-04T12:13:04.276 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:04 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-04T12:13:04.444 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:04 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-05-04T12:13:04.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:04 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-04T12:13:05.344 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:05 smithi146 ceph-mon[35311]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-05-04T12:13:05.345 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:05 smithi146 ceph-mon[35311]: Deploying daemon osd.4 on smithi146 2024-05-04T12:13:05.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:05 smithi079 ceph-mon[27222]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-05-04T12:13:05.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:05 smithi079 ceph-mon[27222]: Deploying daemon osd.4 on smithi146 2024-05-04T12:13:07.164 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:07 smithi146 ceph-mon[35311]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-05-04T12:13:07.194 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:07 smithi079 ceph-mon[27222]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-05-04T12:13:09.444 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:09 smithi079 ceph-mon[27222]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-05-04T12:13:09.526 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:09 smithi146 ceph-mon[35311]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-05-04T12:13:10.526 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:10 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:13:10.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:10 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:13:10.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:10 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-04T12:13:10.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:10 smithi146 ceph-mon[35311]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-05-04T12:13:10.695 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:10 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:13:10.695 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:10 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:13:10.695 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:10 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-04T12:13:10.695 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:10 smithi079 ceph-mon[27222]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-05-04T12:13:12.195 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:12 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-04T12:13:12.276 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:12 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-04T12:13:13.205 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:13:13.207+0000 7f0719ffb700 1 -- 172.21.15.146:0/1338561717 <== mgr.14219 v2:172.21.15.79:6800/402764217 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f071c19ea80 con 0x7f070c06c4e0 2024-05-04T12:13:13.206 INFO:teuthology.orchestra.run.smithi146.stdout:Created osd(s) 4 on host 'smithi146' 2024-05-04T12:13:13.208 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:13:13.210+0000 7f0722ce1700 1 -- 172.21.15.146:0/1338561717 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f070c06c4e0 msgr2=0x7f070c06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:13:13.208 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:13:13.210+0000 7f0722ce1700 1 --2- 172.21.15.146:0/1338561717 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f070c06c4e0 0x7f070c06e9a0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f0710005090 tx=0x7f071000b540 comp rx=0 tx=0).stop 2024-05-04T12:13:13.208 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:13:13.210+0000 7f0722ce1700 1 -- 172.21.15.146:0/1338561717 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f071c102da0 msgr2=0x7f071c198cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:13:13.208 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:13:13.210+0000 7f0722ce1700 1 --2- 172.21.15.146:0/1338561717 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f071c102da0 0x7f071c198cb0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f070800b4b0 tx=0x7f070800b7c0 comp rx=0 tx=0).stop 2024-05-04T12:13:13.208 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:13:13.210+0000 7f0722ce1700 1 -- 172.21.15.146:0/1338561717 shutdown_connections 2024-05-04T12:13:13.208 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:13:13.210+0000 7f0722ce1700 1 --2- 172.21.15.146:0/1338561717 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f070c06c4e0 0x7f070c06e9a0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:13:13.208 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:13:13.210+0000 7f0722ce1700 1 --2- 172.21.15.146:0/1338561717 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f071c1003e0 0x7f071c198770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:13:13.208 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:13:13.210+0000 7f0722ce1700 1 --2- 172.21.15.146:0/1338561717 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f071c102da0 0x7f071c198cb0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:13:13.209 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:13:13.210+0000 7f0722ce1700 1 -- 172.21.15.146:0/1338561717 >> 172.21.15.146:0/1338561717 conn(0x7f071c0fa490 msgr2=0x7f071c0ff990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:13:13.209 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:13:13.210+0000 7f0722ce1700 1 -- 172.21.15.146:0/1338561717 shutdown_connections 2024-05-04T12:13:13.209 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:13:13.210+0000 7f0722ce1700 1 -- 172.21.15.146:0/1338561717 wait complete. 2024-05-04T12:13:13.398 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:13 smithi146 ceph-mon[35311]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-05-04T12:13:13.444 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:13 smithi079 ceph-mon[27222]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-05-04T12:13:14.057 DEBUG:teuthology.orchestra.run.smithi146:osd.4> sudo journalctl -f -n 0 -u ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968@osd.4.service 2024-05-04T12:13:14.060 INFO:tasks.cephadm:Deploying osd.5 on smithi146 with /dev/vg_nvme/lv_2... 2024-05-04T12:13:14.060 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d9b985c4-0a0e-11ef-bc97-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-05-04T12:13:14.307 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/d9b985c4-0a0e-11ef-bc97-c7b262605968/mon.smithi146/config 2024-05-04T12:13:14.429 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:14 smithi146 ceph-mon[35311]: from='osd.4 [v2:172.21.15.146:6808/3913803877,v1:172.21.15.146:6809/3913803877]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-04T12:13:14.429 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:14 smithi146 ceph-mon[35311]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-04T12:13:14.430 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:14 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:13:14.430 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:14 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:13:14.444 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:14 smithi079 ceph-mon[27222]: from='osd.4 [v2:172.21.15.146:6808/3913803877,v1:172.21.15.146:6809/3913803877]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-04T12:13:14.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:14 smithi079 ceph-mon[27222]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-04T12:13:14.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:14 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:13:14.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:14 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:13:15.387 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:15 smithi146 ceph-mon[35311]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-05-04T12:13:15.388 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:15 smithi146 ceph-mon[35311]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-05-04T12:13:15.388 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:15 smithi146 ceph-mon[35311]: osdmap e32: 5 total, 4 up, 5 in 2024-05-04T12:13:15.388 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:15 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-04T12:13:15.388 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:15 smithi146 ceph-mon[35311]: from='osd.4 [v2:172.21.15.146:6808/3913803877,v1:172.21.15.146:6809/3913803877]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-05-04T12:13:15.388 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:15 smithi146 ceph-mon[35311]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-05-04T12:13:15.388 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:15 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:13:15.388 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:15 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:13:15.388 INFO:journalctl@ceph.osd.4.smithi146.stdout:May 04 12:13:15 smithi146 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-4[47723]: 2024-05-04T12:13:15.048+0000 7f634fdcd700 -1 osd.4 0 waiting for initial osdmap 2024-05-04T12:13:15.388 INFO:journalctl@ceph.osd.4.smithi146.stdout:May 04 12:13:15 smithi146 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-4[47723]: 2024-05-04T12:13:15.056+0000 7f634abc0700 -1 osd.4 33 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-04T12:13:15.444 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:15 smithi079 ceph-mon[27222]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-05-04T12:13:15.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:15 smithi079 ceph-mon[27222]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-05-04T12:13:15.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:15 smithi079 ceph-mon[27222]: osdmap e32: 5 total, 4 up, 5 in 2024-05-04T12:13:15.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:15 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-04T12:13:15.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:15 smithi079 ceph-mon[27222]: from='osd.4 [v2:172.21.15.146:6808/3913803877,v1:172.21.15.146:6809/3913803877]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-05-04T12:13:15.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:15 smithi079 ceph-mon[27222]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-05-04T12:13:15.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:15 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:13:15.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:15 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:13:16.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:16 smithi079 ceph-mon[27222]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]': finished 2024-05-04T12:13:16.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:16 smithi079 ceph-mon[27222]: osdmap e33: 5 total, 4 up, 5 in 2024-05-04T12:13:16.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:16 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-04T12:13:16.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:16 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-04T12:13:16.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:16 smithi079 ceph-mon[27222]: osd.4 [v2:172.21.15.146:6808/3913803877,v1:172.21.15.146:6809/3913803877] boot 2024-05-04T12:13:16.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:16 smithi079 ceph-mon[27222]: osdmap e34: 5 total, 5 up, 5 in 2024-05-04T12:13:16.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:16 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-04T12:13:16.466 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:16 smithi146 ceph-mon[35311]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]': finished 2024-05-04T12:13:16.466 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:16 smithi146 ceph-mon[35311]: osdmap e33: 5 total, 4 up, 5 in 2024-05-04T12:13:16.466 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:16 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-04T12:13:16.466 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:16 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-04T12:13:16.467 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:16 smithi146 ceph-mon[35311]: osd.4 [v2:172.21.15.146:6808/3913803877,v1:172.21.15.146:6809/3913803877] boot 2024-05-04T12:13:16.467 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:16 smithi146 ceph-mon[35311]: osdmap e34: 5 total, 5 up, 5 in 2024-05-04T12:13:16.467 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:16 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-04T12:13:17.224 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:17 smithi146 ceph-mon[35311]: purged_snaps scrub starts 2024-05-04T12:13:17.224 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:17 smithi146 ceph-mon[35311]: purged_snaps scrub ok 2024-05-04T12:13:17.224 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:17 smithi146 ceph-mon[35311]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-05-04T12:13:17.318 INFO:teuthology.orchestra.run.smithi146.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-05-04T12:13:17.318 INFO:teuthology.orchestra.run.smithi146.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-05-04T12:13:17.318 INFO:teuthology.orchestra.run.smithi146.stderr: stderr: 10+0 records in 2024-05-04T12:13:17.318 INFO:teuthology.orchestra.run.smithi146.stderr:10+0 records out 2024-05-04T12:13:17.318 INFO:teuthology.orchestra.run.smithi146.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0124062 s, 845 MB/s 2024-05-04T12:13:17.319 INFO:teuthology.orchestra.run.smithi146.stderr: stderr: 2024-05-04T12:13:17.319 INFO:teuthology.orchestra.run.smithi146.stderr:--> Zapping successful for: 2024-05-04T12:13:17.444 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:17 smithi079 ceph-mon[27222]: purged_snaps scrub starts 2024-05-04T12:13:17.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:17 smithi079 ceph-mon[27222]: purged_snaps scrub ok 2024-05-04T12:13:17.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:17 smithi079 ceph-mon[27222]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-05-04T12:13:18.276 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:18 smithi146 ceph-mon[35311]: osdmap e35: 5 total, 5 up, 5 in 2024-05-04T12:13:18.389 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d9b985c4-0a0e-11ef-bc97-c7b262605968 -- ceph orch daemon add osd smithi146:vg_nvme/lv_2 2024-05-04T12:13:18.444 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:18 smithi079 ceph-mon[27222]: osdmap e35: 5 total, 5 up, 5 in 2024-05-04T12:13:18.921 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/d9b985c4-0a0e-11ef-bc97-c7b262605968/mon.smithi146/config 2024-05-04T12:13:19.093 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:19 smithi146 ceph-mon[35311]: pgmap v116: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-05-04T12:13:19.093 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:19 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:13:19.093 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:19 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:13:19.093 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:19 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-04T12:13:19.093 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:19 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-04T12:13:19.093 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:19 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:13:19.093 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:19 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-04T12:13:19.093 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:19 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-04T12:13:19.093 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:19 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:13:19.093 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:19 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-04T12:13:19.093 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:19 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-04T12:13:19.093 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:19 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-04T12:13:19.093 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:19 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:13:19.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:19 smithi079 ceph-mon[27222]: pgmap v116: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-05-04T12:13:19.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:19 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:13:19.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:19 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:13:19.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:19 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-04T12:13:19.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:19 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-04T12:13:19.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:19 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:13:19.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:19 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-04T12:13:19.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:19 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-04T12:13:19.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:19 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:13:19.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:19 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-04T12:13:19.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:19 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-04T12:13:19.446 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:19 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-04T12:13:19.446 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:19 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:13:20.195 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:20 smithi079 ceph-mon[27222]: Detected new or changed devices on smithi146 2024-05-04T12:13:20.195 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:20 smithi079 ceph-mon[27222]: Adjusting osd_memory_target on smithi146 to 7456M 2024-05-04T12:13:20.371 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:20 smithi146 ceph-mon[35311]: Detected new or changed devices on smithi146 2024-05-04T12:13:20.371 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:20 smithi146 ceph-mon[35311]: Adjusting osd_memory_target on smithi146 to 7456M 2024-05-04T12:13:20.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:13:20.622+0000 7f8514639700 1 -- 172.21.15.146:0/842219838 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f850c101f40 msgr2=0x7f850c1023a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:13:20.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:13:20.622+0000 7f8514639700 1 --2- 172.21.15.146:0/842219838 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f850c101f40 0x7f850c1023a0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f84fc009910 tx=0x7f84fc009c20 comp rx=0 tx=0).stop 2024-05-04T12:13:20.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:13:20.622+0000 7f8514639700 1 -- 172.21.15.146:0/842219838 shutdown_connections 2024-05-04T12:13:20.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:13:20.622+0000 7f8514639700 1 --2- 172.21.15.146:0/842219838 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f850c101f40 0x7f850c1023a0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:13:20.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:13:20.622+0000 7f8514639700 1 --2- 172.21.15.146:0/842219838 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f850c0fef60 0x7f850c0ff380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:13:20.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:13:20.622+0000 7f8514639700 1 -- 172.21.15.146:0/842219838 >> 172.21.15.146:0/842219838 conn(0x7f850c0fa8c0 msgr2=0x7f850c0fcd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:13:20.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:13:20.622+0000 7f8514639700 1 -- 172.21.15.146:0/842219838 shutdown_connections 2024-05-04T12:13:20.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:13:20.622+0000 7f8514639700 1 -- 172.21.15.146:0/842219838 wait complete. 2024-05-04T12:13:20.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:13:20.622+0000 7f8514639700 1 Processor -- start 2024-05-04T12:13:20.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:13:20.622+0000 7f8514639700 1 -- start start 2024-05-04T12:13:20.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:13:20.622+0000 7f8514639700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f850c0fef60 0x7f850c1964b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:13:20.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:13:20.623+0000 7f8514639700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f850c101f40 0x7f850c1969f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:13:20.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:13:20.623+0000 7f8514639700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f850c197010 con 0x7f850c0fef60 2024-05-04T12:13:20.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:13:20.623+0000 7f8514639700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f850c197150 con 0x7f850c101f40 2024-05-04T12:13:20.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:13:20.623+0000 7f8511bd4700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f850c101f40 0x7f850c1969f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:13:20.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:13:20.623+0000 7f8511bd4700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f850c101f40 0x7f850c1969f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:59338/0 (socket says 172.21.15.146:59338) 2024-05-04T12:13:20.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:13:20.623+0000 7f8511bd4700 1 -- 172.21.15.146:0/1062973210 learned_addr learned my addr 172.21.15.146:0/1062973210 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-05-04T12:13:20.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:13:20.623+0000 7f85123d5700 1 --2- 172.21.15.146:0/1062973210 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f850c0fef60 0x7f850c1964b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:13:20.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:13:20.623+0000 7f8511bd4700 1 -- 172.21.15.146:0/1062973210 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f850c0fef60 msgr2=0x7f850c1964b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:13:20.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:13:20.623+0000 7f8511bd4700 1 --2- 172.21.15.146:0/1062973210 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f850c0fef60 0x7f850c1964b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:13:20.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:13:20.623+0000 7f8511bd4700 1 -- 172.21.15.146:0/1062973210 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84fc0095f0 con 0x7f850c101f40 2024-05-04T12:13:20.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:13:20.623+0000 7f8511bd4700 1 --2- 172.21.15.146:0/1062973210 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f850c101f40 0x7f850c1969f0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f84fc006010 tx=0x7f84fc00be20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:13:20.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:13:20.623+0000 7f85037fe700 1 -- 172.21.15.146:0/1062973210 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f84fc01d070 con 0x7f850c101f40 2024-05-04T12:13:20.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:13:20.623+0000 7f85037fe700 1 -- 172.21.15.146:0/1062973210 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f84fc00f460 con 0x7f850c101f40 2024-05-04T12:13:20.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:13:20.623+0000 7f85037fe700 1 -- 172.21.15.146:0/1062973210 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f84fc017630 con 0x7f850c101f40 2024-05-04T12:13:20.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:13:20.623+0000 7f8514639700 1 -- 172.21.15.146:0/1062973210 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f850c19bba0 con 0x7f850c101f40 2024-05-04T12:13:20.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:13:20.623+0000 7f8514639700 1 -- 172.21.15.146:0/1062973210 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f850c19c090 con 0x7f850c101f40 2024-05-04T12:13:20.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:13:20.624+0000 7f8514639700 1 -- 172.21.15.146:0/1062973210 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f850c109800 con 0x7f850c101f40 2024-05-04T12:13:20.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:13:20.625+0000 7f85037fe700 1 -- 172.21.15.146:0/1062973210 <== mon.1 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f84fc003a60 con 0x7f850c101f40 2024-05-04T12:13:20.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:13:20.625+0000 7f85037fe700 1 --2- 172.21.15.146:0/1062973210 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f84f806c4e0 0x7f84f806e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:13:20.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:13:20.626+0000 7f85037fe700 1 -- 172.21.15.146:0/1062973210 <== mon.1 v2:172.21.15.146:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 4206+0+0 (secure 0 0 0) 0x7f84fc08c760 con 0x7f850c101f40 2024-05-04T12:13:20.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:13:20.626+0000 7f85123d5700 1 --2- 172.21.15.146:0/1062973210 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f84f806c4e0 0x7f84f806e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:13:20.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:13:20.626+0000 7f85123d5700 1 --2- 172.21.15.146:0/1062973210 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f84f806c4e0 0x7f84f806e9a0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f8508005950 tx=0x7f85080058e0 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:13:20.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:13:20.630+0000 7f85037fe700 1 -- 172.21.15.146:0/1062973210 <== mon.1 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f84fc05aff0 con 0x7f850c101f40 2024-05-04T12:13:20.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:13:20.823+0000 7f8514639700 1 -- 172.21.15.146:0/1062973210 --> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi146:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f850c106890 con 0x7f84f806c4e0 2024-05-04T12:13:21.326 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:21 smithi146 ceph-mon[35311]: pgmap v117: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-05-04T12:13:21.326 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:21 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-04T12:13:21.326 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:21 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-04T12:13:21.326 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:21 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-04T12:13:21.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:21 smithi079 ceph-mon[27222]: pgmap v117: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-05-04T12:13:21.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:21 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-04T12:13:21.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:21 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-04T12:13:21.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:21 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-04T12:13:22.264 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:22 smithi146 ceph-mon[35311]: from='client.24169 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi146:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-04T12:13:22.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:22 smithi079 ceph-mon[27222]: from='client.24169 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi146:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-04T12:13:23.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:23 smithi079 ceph-mon[27222]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-05-04T12:13:23.526 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:23 smithi146 ceph-mon[35311]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-05-04T12:13:24.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:24 smithi079 ceph-mon[27222]: from='client.? 172.21.15.146:0/1340874069' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "02174a1e-ba23-4235-bd22-0969c319b052"}]: dispatch 2024-05-04T12:13:24.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:24 smithi079 ceph-mon[27222]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "02174a1e-ba23-4235-bd22-0969c319b052"}]: dispatch 2024-05-04T12:13:24.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:24 smithi079 ceph-mon[27222]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "02174a1e-ba23-4235-bd22-0969c319b052"}]': finished 2024-05-04T12:13:24.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:24 smithi079 ceph-mon[27222]: osdmap e36: 6 total, 5 up, 6 in 2024-05-04T12:13:24.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:24 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-04T12:13:24.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:24 smithi079 ceph-mon[27222]: from='client.? 172.21.15.146:0/365034121' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-04T12:13:24.526 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:24 smithi146 ceph-mon[35311]: from='client.? 172.21.15.146:0/1340874069' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "02174a1e-ba23-4235-bd22-0969c319b052"}]: dispatch 2024-05-04T12:13:24.526 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:24 smithi146 ceph-mon[35311]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "02174a1e-ba23-4235-bd22-0969c319b052"}]: dispatch 2024-05-04T12:13:24.526 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:24 smithi146 ceph-mon[35311]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "02174a1e-ba23-4235-bd22-0969c319b052"}]': finished 2024-05-04T12:13:24.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:24 smithi146 ceph-mon[35311]: osdmap e36: 6 total, 5 up, 6 in 2024-05-04T12:13:24.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:24 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-04T12:13:24.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:24 smithi146 ceph-mon[35311]: from='client.? 172.21.15.146:0/365034121' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-04T12:13:25.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:25 smithi079 ceph-mon[27222]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-05-04T12:13:25.526 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:25 smithi146 ceph-mon[35311]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-05-04T12:13:27.248 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:27 smithi146 ceph-mon[35311]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-05-04T12:13:27.248 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:27 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-04T12:13:27.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:27 smithi079 ceph-mon[27222]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-05-04T12:13:27.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:27 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-04T12:13:29.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:29 smithi079 ceph-mon[27222]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-05-04T12:13:29.521 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:29 smithi146 ceph-mon[35311]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-05-04T12:13:31.276 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:31 smithi146 ceph-mon[35311]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-05-04T12:13:31.276 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:31 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-05-04T12:13:31.276 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:31 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-04T12:13:31.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:31 smithi079 ceph-mon[27222]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-05-04T12:13:31.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:31 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-05-04T12:13:31.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:31 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-04T12:13:32.276 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:32 smithi146 ceph-mon[35311]: Deploying daemon osd.5 on smithi146 2024-05-04T12:13:32.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:32 smithi079 ceph-mon[27222]: Deploying daemon osd.5 on smithi146 2024-05-04T12:13:33.276 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:33 smithi146 ceph-mon[35311]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-05-04T12:13:33.444 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:33 smithi079 ceph-mon[27222]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-05-04T12:13:35.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:35 smithi079 ceph-mon[27222]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-05-04T12:13:35.478 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:35 smithi146 ceph-mon[35311]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-05-04T12:13:37.276 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:37 smithi146 ceph-mon[35311]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-05-04T12:13:37.276 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:37 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:13:37.276 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:37 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:13:37.277 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:37 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-04T12:13:37.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:37 smithi079 ceph-mon[27222]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-05-04T12:13:37.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:37 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:13:37.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:37 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:13:37.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:37 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-04T12:13:39.276 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:39 smithi146 ceph-mon[35311]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-05-04T12:13:39.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:39 smithi079 ceph-mon[27222]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-05-04T12:13:39.735 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:13:39.736+0000 7f85037fe700 1 -- 172.21.15.146:0/1062973210 <== mgr.14219 v2:172.21.15.79:6800/402764217 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f850c106890 con 0x7f84f806c4e0 2024-05-04T12:13:39.735 INFO:teuthology.orchestra.run.smithi146.stdout:Created osd(s) 5 on host 'smithi146' 2024-05-04T12:13:39.737 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:13:39.738+0000 7f8514639700 1 -- 172.21.15.146:0/1062973210 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f84f806c4e0 msgr2=0x7f84f806e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:13:39.738 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:13:39.738+0000 7f8514639700 1 --2- 172.21.15.146:0/1062973210 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f84f806c4e0 0x7f84f806e9a0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f8508005950 tx=0x7f85080058e0 comp rx=0 tx=0).stop 2024-05-04T12:13:39.738 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:13:39.738+0000 7f8514639700 1 -- 172.21.15.146:0/1062973210 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f850c101f40 msgr2=0x7f850c1969f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:13:39.738 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:13:39.738+0000 7f8514639700 1 --2- 172.21.15.146:0/1062973210 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f850c101f40 0x7f850c1969f0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f84fc006010 tx=0x7f84fc00be20 comp rx=0 tx=0).stop 2024-05-04T12:13:39.738 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:13:39.738+0000 7f8514639700 1 -- 172.21.15.146:0/1062973210 shutdown_connections 2024-05-04T12:13:39.738 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:13:39.738+0000 7f8514639700 1 --2- 172.21.15.146:0/1062973210 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f84f806c4e0 0x7f84f806e9a0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:13:39.738 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:13:39.738+0000 7f8514639700 1 --2- 172.21.15.146:0/1062973210 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f850c0fef60 0x7f850c1964b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:13:39.738 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:13:39.738+0000 7f8514639700 1 --2- 172.21.15.146:0/1062973210 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f850c101f40 0x7f850c1969f0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:13:39.738 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:13:39.738+0000 7f8514639700 1 -- 172.21.15.146:0/1062973210 >> 172.21.15.146:0/1062973210 conn(0x7f850c0fa8c0 msgr2=0x7f850c105170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:13:39.738 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:13:39.738+0000 7f8514639700 1 -- 172.21.15.146:0/1062973210 shutdown_connections 2024-05-04T12:13:39.738 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:13:39.738+0000 7f8514639700 1 -- 172.21.15.146:0/1062973210 wait complete. 2024-05-04T12:13:40.886 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:40 smithi146 ceph-mon[35311]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-05-04T12:13:40.886 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:40 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:13:40.886 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:40 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:13:40.886 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:40 smithi146 ceph-mon[35311]: from='osd.5 [v2:172.21.15.146:6816/3191460146,v1:172.21.15.146:6817/3191460146]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-04T12:13:40.886 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:40 smithi146 ceph-mon[35311]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-04T12:13:40.887 DEBUG:teuthology.orchestra.run.smithi146:osd.5> sudo journalctl -f -n 0 -u ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968@osd.5.service 2024-05-04T12:13:40.890 INFO:tasks.cephadm:Waiting for 6 OSDs to come up... 2024-05-04T12:13:40.890 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d9b985c4-0a0e-11ef-bc97-c7b262605968 -- ceph osd stat -f json 2024-05-04T12:13:40.945 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:40 smithi079 ceph-mon[27222]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-05-04T12:13:40.945 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:40 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:13:40.945 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:40 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:13:40.945 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:40 smithi079 ceph-mon[27222]: from='osd.5 [v2:172.21.15.146:6816/3191460146,v1:172.21.15.146:6817/3191460146]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-04T12:13:40.945 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:40 smithi079 ceph-mon[27222]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-04T12:13:41.111 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/d9b985c4-0a0e-11ef-bc97-c7b262605968/mon.smithi079/config 2024-05-04T12:13:42.296 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:42 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:13:42.296 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:42 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:13:42.296 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:42 smithi146 ceph-mon[35311]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-05-04T12:13:42.296 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:42 smithi146 ceph-mon[35311]: osdmap e37: 6 total, 5 up, 6 in 2024-05-04T12:13:42.296 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:42 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-04T12:13:42.296 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:42 smithi146 ceph-mon[35311]: from='osd.5 [v2:172.21.15.146:6816/3191460146,v1:172.21.15.146:6817/3191460146]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-05-04T12:13:42.297 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:42 smithi146 ceph-mon[35311]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-05-04T12:13:42.297 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:42 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-04T12:13:42.297 INFO:journalctl@ceph.osd.5.smithi146.stdout:May 04 12:13:42 smithi146 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-5[54242]: 2024-05-04T12:13:42.146+0000 7fd480375700 -1 osd.5 0 waiting for initial osdmap 2024-05-04T12:13:42.297 INFO:journalctl@ceph.osd.5.smithi146.stdout:May 04 12:13:42 smithi146 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-5[54242]: 2024-05-04T12:13:42.154+0000 7fd47b168700 -1 osd.5 38 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-04T12:13:42.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:42 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:13:42.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:42 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:13:42.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:42 smithi079 ceph-mon[27222]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-05-04T12:13:42.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:42 smithi079 ceph-mon[27222]: osdmap e37: 6 total, 5 up, 6 in 2024-05-04T12:13:42.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:42 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-04T12:13:42.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:42 smithi079 ceph-mon[27222]: from='osd.5 [v2:172.21.15.146:6816/3191460146,v1:172.21.15.146:6817/3191460146]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-05-04T12:13:42.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:42 smithi079 ceph-mon[27222]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-05-04T12:13:42.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:42 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-04T12:13:42.810 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:42.809+0000 7f92a2669700 1 -- 172.21.15.79:0/1979647511 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f929c100da0 msgr2=0x7f929c101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:13:42.810 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:42.809+0000 7f92a2669700 1 --2- 172.21.15.79:0/1979647511 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f929c100da0 0x7f929c101200 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f9290009910 tx=0x7f9290009c20 comp rx=0 tx=0).stop 2024-05-04T12:13:42.810 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:42.809+0000 7f92a2669700 1 -- 172.21.15.79:0/1979647511 shutdown_connections 2024-05-04T12:13:42.810 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:42.809+0000 7f92a2669700 1 --2- 172.21.15.79:0/1979647511 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f929c100da0 0x7f929c101200 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:13:42.810 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:42.809+0000 7f92a2669700 1 --2- 172.21.15.79:0/1979647511 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f929c0ffba0 0x7f929c0fffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:13:42.810 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:42.809+0000 7f92a2669700 1 -- 172.21.15.79:0/1979647511 >> 172.21.15.79:0/1979647511 conn(0x7f929c0fb120 msgr2=0x7f929c0fd580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:13:42.810 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:42.809+0000 7f92a2669700 1 -- 172.21.15.79:0/1979647511 shutdown_connections 2024-05-04T12:13:42.811 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:42.809+0000 7f92a2669700 1 -- 172.21.15.79:0/1979647511 wait complete. 2024-05-04T12:13:42.811 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:42.810+0000 7f92a2669700 1 Processor -- start 2024-05-04T12:13:42.811 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:42.810+0000 7f92a2669700 1 -- start start 2024-05-04T12:13:42.811 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:42.810+0000 7f92a2669700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f929c0ffba0 0x7f929c069970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:13:42.811 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:42.810+0000 7f92a2669700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f929c100da0 0x7f929c19b200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:13:42.811 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:42.810+0000 7f92a2669700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f929c19b7d0 con 0x7f929c100da0 2024-05-04T12:13:42.811 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:42.810+0000 7f92a2669700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f929c19b910 con 0x7f929c0ffba0 2024-05-04T12:13:42.811 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:42.810+0000 7f929b7fe700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f929c100da0 0x7f929c19b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:13:42.811 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:42.810+0000 7f929b7fe700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f929c100da0 0x7f929c19b200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:34940/0 (socket says 172.21.15.79:34940) 2024-05-04T12:13:42.811 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:42.810+0000 7f929b7fe700 1 -- 172.21.15.79:0/150521525 learned_addr learned my addr 172.21.15.79:0/150521525 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:13:42.811 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:42.810+0000 7f929bfff700 1 --2- 172.21.15.79:0/150521525 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f929c0ffba0 0x7f929c069970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:13:42.811 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:42.810+0000 7f929b7fe700 1 -- 172.21.15.79:0/150521525 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f929c0ffba0 msgr2=0x7f929c069970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:13:42.811 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:42.810+0000 7f929b7fe700 1 --2- 172.21.15.79:0/150521525 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f929c0ffba0 0x7f929c069970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:13:42.811 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:42.810+0000 7f929b7fe700 1 -- 172.21.15.79:0/150521525 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92900095f0 con 0x7f929c100da0 2024-05-04T12:13:42.812 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:42.811+0000 7f929b7fe700 1 --2- 172.21.15.79:0/150521525 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f929c100da0 0x7f929c19b200 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f9290006010 tx=0x7f929000bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:13:42.812 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:42.811+0000 7f92997fa700 1 -- 172.21.15.79:0/150521525 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f929001d070 con 0x7f929c100da0 2024-05-04T12:13:42.812 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:42.811+0000 7f92a2669700 1 -- 172.21.15.79:0/150521525 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f929c19bb90 con 0x7f929c100da0 2024-05-04T12:13:42.812 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:42.811+0000 7f92997fa700 1 -- 172.21.15.79:0/150521525 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f929000f460 con 0x7f929c100da0 2024-05-04T12:13:42.812 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:42.811+0000 7f92997fa700 1 -- 172.21.15.79:0/150521525 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9290021640 con 0x7f929c100da0 2024-05-04T12:13:42.812 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:42.811+0000 7f92a2669700 1 -- 172.21.15.79:0/150521525 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f929c19c080 con 0x7f929c100da0 2024-05-04T12:13:42.813 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:42.812+0000 7f92a2669700 1 -- 172.21.15.79:0/150521525 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f929c108700 con 0x7f929c100da0 2024-05-04T12:13:42.813 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:42.812+0000 7f92997fa700 1 -- 172.21.15.79:0/150521525 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f9290003a60 con 0x7f929c100da0 2024-05-04T12:13:42.813 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:42.813+0000 7f92997fa700 1 --2- 172.21.15.79:0/150521525 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f928406c4e0 0x7f928406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:13:42.813 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:42.813+0000 7f92997fa700 1 -- 172.21.15.79:0/150521525 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4358+0+0 (secure 0 0 0) 0x7f929008c620 con 0x7f929c100da0 2024-05-04T12:13:42.813 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:42.813+0000 7f929bfff700 1 --2- 172.21.15.79:0/150521525 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f928406c4e0 0x7f928406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:13:42.813 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:42.813+0000 7f929bfff700 1 --2- 172.21.15.79:0/150521525 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f928406c4e0 0x7f928406e9a0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f929c100c00 tx=0x7f928c00b410 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:13:42.816 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:42.817+0000 7f92997fa700 1 -- 172.21.15.79:0/150521525 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f929005ace0 con 0x7f929c100da0 2024-05-04T12:13:43.003 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:43.004+0000 7f92a2669700 1 -- 172.21.15.79:0/150521525 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f929c19c690 con 0x7f929c100da0 2024-05-04T12:13:43.004 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:43.005+0000 7f92997fa700 1 -- 172.21.15.79:0/150521525 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v38) v1 ==== 74+0+130 (secure 0 0 0) 0x7f929005a870 con 0x7f929c100da0 2024-05-04T12:13:43.004 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-05-04T12:13:43.006 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:43.007+0000 7f92a2669700 1 -- 172.21.15.79:0/150521525 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f928406c4e0 msgr2=0x7f928406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:13:43.006 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:43.007+0000 7f92a2669700 1 --2- 172.21.15.79:0/150521525 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f928406c4e0 0x7f928406e9a0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f929c100c00 tx=0x7f928c00b410 comp rx=0 tx=0).stop 2024-05-04T12:13:43.006 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:43.007+0000 7f92a2669700 1 -- 172.21.15.79:0/150521525 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f929c100da0 msgr2=0x7f929c19b200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:13:43.006 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:43.007+0000 7f92a2669700 1 --2- 172.21.15.79:0/150521525 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f929c100da0 0x7f929c19b200 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f9290006010 tx=0x7f929000bd70 comp rx=0 tx=0).stop 2024-05-04T12:13:43.006 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:43.007+0000 7f92a2669700 1 -- 172.21.15.79:0/150521525 shutdown_connections 2024-05-04T12:13:43.006 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:43.007+0000 7f92a2669700 1 --2- 172.21.15.79:0/150521525 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f928406c4e0 0x7f928406e9a0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:13:43.006 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:43.007+0000 7f92a2669700 1 --2- 172.21.15.79:0/150521525 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f929c0ffba0 0x7f929c069970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:13:43.007 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:43.007+0000 7f92a2669700 1 --2- 172.21.15.79:0/150521525 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f929c100da0 0x7f929c19b200 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:13:43.007 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:43.007+0000 7f92a2669700 1 -- 172.21.15.79:0/150521525 >> 172.21.15.79:0/150521525 conn(0x7f929c0fb120 msgr2=0x7f929c103fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:13:43.007 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:43.007+0000 7f92a2669700 1 -- 172.21.15.79:0/150521525 shutdown_connections 2024-05-04T12:13:43.007 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:43.007+0000 7f92a2669700 1 -- 172.21.15.79:0/150521525 wait complete. 2024-05-04T12:13:43.276 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:43 smithi146 ceph-mon[35311]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-05-04T12:13:43.277 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:43 smithi146 ceph-mon[35311]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]': finished 2024-05-04T12:13:43.277 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:43 smithi146 ceph-mon[35311]: osdmap e38: 6 total, 5 up, 6 in 2024-05-04T12:13:43.277 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:43 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-04T12:13:43.277 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:43 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-04T12:13:43.277 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:43 smithi146 ceph-mon[35311]: from='client.? 172.21.15.79:0/150521525' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-05-04T12:13:43.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:43 smithi079 ceph-mon[27222]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-05-04T12:13:43.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:43 smithi079 ceph-mon[27222]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]': finished 2024-05-04T12:13:43.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:43 smithi079 ceph-mon[27222]: osdmap e38: 6 total, 5 up, 6 in 2024-05-04T12:13:43.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:43 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-04T12:13:43.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:43 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-04T12:13:43.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:43 smithi079 ceph-mon[27222]: from='client.? 172.21.15.79:0/150521525' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-05-04T12:13:43.677 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":38,"num_osds":6,"num_up_osds":5,"osd_up_since":1714824796,"num_in_osds":6,"osd_in_since":1714824803,"num_remapped_pgs":0} 2024-05-04T12:13:44.389 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:44 smithi146 ceph-mon[35311]: purged_snaps scrub starts 2024-05-04T12:13:44.389 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:44 smithi146 ceph-mon[35311]: purged_snaps scrub ok 2024-05-04T12:13:44.389 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:44 smithi146 ceph-mon[35311]: osd.5 [v2:172.21.15.146:6816/3191460146,v1:172.21.15.146:6817/3191460146] boot 2024-05-04T12:13:44.389 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:44 smithi146 ceph-mon[35311]: osdmap e39: 6 total, 6 up, 6 in 2024-05-04T12:13:44.389 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:44 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-04T12:13:44.389 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:44 smithi146 ceph-mon[35311]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-05-04T12:13:44.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:44 smithi079 ceph-mon[27222]: purged_snaps scrub starts 2024-05-04T12:13:44.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:44 smithi079 ceph-mon[27222]: purged_snaps scrub ok 2024-05-04T12:13:44.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:44 smithi079 ceph-mon[27222]: osd.5 [v2:172.21.15.146:6816/3191460146,v1:172.21.15.146:6817/3191460146] boot 2024-05-04T12:13:44.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:44 smithi079 ceph-mon[27222]: osdmap e39: 6 total, 6 up, 6 in 2024-05-04T12:13:44.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:44 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-04T12:13:44.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:44 smithi079 ceph-mon[27222]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-05-04T12:13:44.678 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d9b985c4-0a0e-11ef-bc97-c7b262605968 -- ceph osd stat -f json 2024-05-04T12:13:44.892 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/d9b985c4-0a0e-11ef-bc97-c7b262605968/mon.smithi079/config 2024-05-04T12:13:45.334 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:45 smithi079 ceph-mon[27222]: osdmap e40: 6 total, 6 up, 6 in 2024-05-04T12:13:45.334 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:45 smithi079 ceph-mon[27222]: Detected new or changed devices on smithi146 2024-05-04T12:13:45.334 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:45 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:13:45.334 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:45 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:13:45.334 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:45 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-04T12:13:45.334 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:45 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-04T12:13:45.334 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:45 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-04T12:13:45.334 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:45 smithi079 ceph-mon[27222]: Adjusting osd_memory_target on smithi146 to 4970M 2024-05-04T12:13:45.334 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:45 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:13:45.334 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:45 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-04T12:13:45.335 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:45 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-04T12:13:45.335 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:45 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:13:45.335 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:45 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-04T12:13:45.335 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:45 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-04T12:13:45.335 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:45 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-04T12:13:45.335 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:45 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:13:45.526 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:45 smithi146 ceph-mon[35311]: osdmap e40: 6 total, 6 up, 6 in 2024-05-04T12:13:45.526 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:45 smithi146 ceph-mon[35311]: Detected new or changed devices on smithi146 2024-05-04T12:13:45.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:45 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:13:45.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:45 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:13:45.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:45 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-04T12:13:45.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:45 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-04T12:13:45.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:45 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-04T12:13:45.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:45 smithi146 ceph-mon[35311]: Adjusting osd_memory_target on smithi146 to 4970M 2024-05-04T12:13:45.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:45 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:13:45.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:45 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-04T12:13:45.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:45 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-04T12:13:45.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:45 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:13:45.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:45 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-04T12:13:45.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:45 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-04T12:13:45.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:45 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-04T12:13:45.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:45 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:13:46.616 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:46.543+0000 7fbcd535f700 1 -- 172.21.15.79:0/3474516040 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbcd0100cb0 msgr2=0x7fbcd01010d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:13:46.616 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:46.543+0000 7fbcd535f700 1 --2- 172.21.15.79:0/3474516040 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbcd0100cb0 0x7fbcd01010d0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fbcc0009910 tx=0x7fbcc0009c20 comp rx=0 tx=0).stop 2024-05-04T12:13:46.616 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:46.543+0000 7fbcd535f700 1 -- 172.21.15.79:0/3474516040 shutdown_connections 2024-05-04T12:13:46.616 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:46.543+0000 7fbcd535f700 1 --2- 172.21.15.79:0/3474516040 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbcd0101eb0 0x7fbcd0102310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:13:46.617 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:46.543+0000 7fbcd535f700 1 --2- 172.21.15.79:0/3474516040 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbcd0100cb0 0x7fbcd01010d0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:13:46.617 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:46.543+0000 7fbcd535f700 1 -- 172.21.15.79:0/3474516040 >> 172.21.15.79:0/3474516040 conn(0x7fbcd00fc230 msgr2=0x7fbcd00fe690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:13:46.617 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:46.543+0000 7fbcd535f700 1 -- 172.21.15.79:0/3474516040 shutdown_connections 2024-05-04T12:13:46.617 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:46.543+0000 7fbcd535f700 1 -- 172.21.15.79:0/3474516040 wait complete. 2024-05-04T12:13:46.617 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:46.543+0000 7fbcd535f700 1 Processor -- start 2024-05-04T12:13:46.617 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:46.544+0000 7fbcd535f700 1 -- start start 2024-05-04T12:13:46.617 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:46.544+0000 7fbcd535f700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbcd0100cb0 0x7fbcd0074b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:13:46.617 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:46.544+0000 7fbcd535f700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbcd0101eb0 0x7fbcd0073180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:13:46.617 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:46.544+0000 7fbcd535f700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcd00736c0 con 0x7fbcd0101eb0 2024-05-04T12:13:46.617 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:46.544+0000 7fbcd535f700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcd0073800 con 0x7fbcd0100cb0 2024-05-04T12:13:46.617 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:46.544+0000 7fbcce7fc700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbcd0101eb0 0x7fbcd0073180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:13:46.617 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:46.544+0000 7fbcce7fc700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbcd0101eb0 0x7fbcd0073180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:51318/0 (socket says 172.21.15.79:51318) 2024-05-04T12:13:46.617 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:46.544+0000 7fbcce7fc700 1 -- 172.21.15.79:0/1163788231 learned_addr learned my addr 172.21.15.79:0/1163788231 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:13:46.617 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:46.544+0000 7fbcceffd700 1 --2- 172.21.15.79:0/1163788231 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbcd0100cb0 0x7fbcd0074b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:13:46.617 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:46.544+0000 7fbcce7fc700 1 -- 172.21.15.79:0/1163788231 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbcd0100cb0 msgr2=0x7fbcd0074b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:13:46.618 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:46.544+0000 7fbcce7fc700 1 --2- 172.21.15.79:0/1163788231 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbcd0100cb0 0x7fbcd0074b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:13:46.618 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:46.544+0000 7fbcce7fc700 1 -- 172.21.15.79:0/1163788231 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcc00095f0 con 0x7fbcd0101eb0 2024-05-04T12:13:46.618 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:46.544+0000 7fbcceffd700 1 --2- 172.21.15.79:0/1163788231 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbcd0100cb0 0x7fbcd0074b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-04T12:13:46.618 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:46.544+0000 7fbcce7fc700 1 --2- 172.21.15.79:0/1163788231 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbcd0101eb0 0x7fbcd0073180 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fbcc400c930 tx=0x7fbcc400ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:13:46.618 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:46.544+0000 7fbcb7fff700 1 -- 172.21.15.79:0/1163788231 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbcc4007ad0 con 0x7fbcd0101eb0 2024-05-04T12:13:46.618 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:46.545+0000 7fbcd535f700 1 -- 172.21.15.79:0/1163788231 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcd0073ae0 con 0x7fbcd0101eb0 2024-05-04T12:13:46.618 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:46.545+0000 7fbcb7fff700 1 -- 172.21.15.79:0/1163788231 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fbcc4007c30 con 0x7fbcd0101eb0 2024-05-04T12:13:46.618 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:46.545+0000 7fbcb7fff700 1 -- 172.21.15.79:0/1163788231 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbcc4018630 con 0x7fbcd0101eb0 2024-05-04T12:13:46.618 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:46.545+0000 7fbcd535f700 1 -- 172.21.15.79:0/1163788231 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcd019e520 con 0x7fbcd0101eb0 2024-05-04T12:13:46.618 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:46.546+0000 7fbcd535f700 1 -- 172.21.15.79:0/1163788231 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcd0109810 con 0x7fbcd0101eb0 2024-05-04T12:13:46.618 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:46.546+0000 7fbcb7fff700 1 -- 172.21.15.79:0/1163788231 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7fbcc400f450 con 0x7fbcd0101eb0 2024-05-04T12:13:46.618 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:46.546+0000 7fbcb7fff700 1 --2- 172.21.15.79:0/1163788231 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7fbcb806c4e0 0x7fbcb806e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:13:46.618 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:46.547+0000 7fbcb7fff700 1 -- 172.21.15.79:0/1163788231 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fbcc408b6c0 con 0x7fbcd0101eb0 2024-05-04T12:13:46.618 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:46.547+0000 7fbcceffd700 1 --2- 172.21.15.79:0/1163788231 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7fbcb806c4e0 0x7fbcb806e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:13:46.619 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:46.547+0000 7fbcceffd700 1 --2- 172.21.15.79:0/1163788231 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7fbcb806c4e0 0x7fbcb806e9a0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fbcc0006010 tx=0x7fbcc0005960 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:13:46.619 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:46.551+0000 7fbcb7fff700 1 -- 172.21.15.79:0/1163788231 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fbcc4059c70 con 0x7fbcd0101eb0 2024-05-04T12:13:46.741 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:46.742+0000 7fbcd535f700 1 -- 172.21.15.79:0/1163788231 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7fbcd019ecb0 con 0x7fbcd0101eb0 2024-05-04T12:13:46.742 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:46.742+0000 7fbcb7fff700 1 -- 172.21.15.79:0/1163788231 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v40) v1 ==== 74+0+130 (secure 0 0 0) 0x7fbcc4059800 con 0x7fbcd0101eb0 2024-05-04T12:13:46.742 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-05-04T12:13:46.743 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:46.745+0000 7fbcd535f700 1 -- 172.21.15.79:0/1163788231 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7fbcb806c4e0 msgr2=0x7fbcb806e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:13:46.744 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:46.745+0000 7fbcd535f700 1 --2- 172.21.15.79:0/1163788231 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7fbcb806c4e0 0x7fbcb806e9a0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fbcc0006010 tx=0x7fbcc0005960 comp rx=0 tx=0).stop 2024-05-04T12:13:46.744 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:46.745+0000 7fbcd535f700 1 -- 172.21.15.79:0/1163788231 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbcd0101eb0 msgr2=0x7fbcd0073180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:13:46.744 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:46.745+0000 7fbcd535f700 1 --2- 172.21.15.79:0/1163788231 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbcd0101eb0 0x7fbcd0073180 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fbcc400c930 tx=0x7fbcc400ccf0 comp rx=0 tx=0).stop 2024-05-04T12:13:46.744 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:46.745+0000 7fbcd535f700 1 -- 172.21.15.79:0/1163788231 shutdown_connections 2024-05-04T12:13:46.744 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:46.745+0000 7fbcd535f700 1 --2- 172.21.15.79:0/1163788231 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7fbcb806c4e0 0x7fbcb806e9a0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:13:46.744 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:46.745+0000 7fbcd535f700 1 --2- 172.21.15.79:0/1163788231 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbcd0100cb0 0x7fbcd0074b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:13:46.744 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:46.745+0000 7fbcd535f700 1 --2- 172.21.15.79:0/1163788231 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbcd0101eb0 0x7fbcd0073180 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:13:46.744 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:46.745+0000 7fbcd535f700 1 -- 172.21.15.79:0/1163788231 >> 172.21.15.79:0/1163788231 conn(0x7fbcd00fc230 msgr2=0x7fbcd01050e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:13:46.744 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:46.745+0000 7fbcd535f700 1 -- 172.21.15.79:0/1163788231 shutdown_connections 2024-05-04T12:13:46.744 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:46.745+0000 7fbcd535f700 1 -- 172.21.15.79:0/1163788231 wait complete. 2024-05-04T12:13:47.420 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:47 smithi079 ceph-mon[27222]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-04T12:13:47.420 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:47 smithi079 ceph-mon[27222]: from='client.? 172.21.15.79:0/1163788231' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-05-04T12:13:47.443 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":40,"num_osds":6,"num_up_osds":6,"osd_up_since":1714824823,"num_in_osds":6,"osd_in_since":1714824803,"num_remapped_pgs":0} 2024-05-04T12:13:47.443 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid d9b985c4-0a0e-11ef-bc97-c7b262605968 -- ceph osd dump --format=json 2024-05-04T12:13:47.526 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:47 smithi146 ceph-mon[35311]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-04T12:13:47.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:47 smithi146 ceph-mon[35311]: from='client.? 172.21.15.79:0/1163788231' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-05-04T12:13:47.650 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/d9b985c4-0a0e-11ef-bc97-c7b262605968/mon.smithi079/config 2024-05-04T12:13:49.072 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:49 smithi079 ceph-mon[27222]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-04T12:13:49.266 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:49.266+0000 7f72c6929700 1 -- 172.21.15.79:0/1587047433 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f72c0101060 msgr2=0x7f72c01047b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:13:49.266 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:49.266+0000 7f72c6929700 1 --2- 172.21.15.79:0/1587047433 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f72c0101060 0x7f72c01047b0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f72b4009910 tx=0x7f72b4009c20 comp rx=0 tx=0).stop 2024-05-04T12:13:49.266 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:49.266+0000 7f72c6929700 1 -- 172.21.15.79:0/1587047433 shutdown_connections 2024-05-04T12:13:49.267 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:49.266+0000 7f72c6929700 1 --2- 172.21.15.79:0/1587047433 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f72c0101060 0x7f72c01047b0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:13:49.267 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:49.266+0000 7f72c6929700 1 --2- 172.21.15.79:0/1587047433 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f72c01006b0 0x7f72c0100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:13:49.267 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:49.266+0000 7f72c6929700 1 -- 172.21.15.79:0/1587047433 >> 172.21.15.79:0/1587047433 conn(0x7f72c0074e30 msgr2=0x7f72c0075240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:13:49.267 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:49.266+0000 7f72c6929700 1 -- 172.21.15.79:0/1587047433 shutdown_connections 2024-05-04T12:13:49.267 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:49.266+0000 7f72c6929700 1 -- 172.21.15.79:0/1587047433 wait complete. 2024-05-04T12:13:49.267 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:49.266+0000 7f72c6929700 1 Processor -- start 2024-05-04T12:13:49.267 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:49.266+0000 7f72c6929700 1 -- start start 2024-05-04T12:13:49.267 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:49.267+0000 7f72c6929700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f72c01006b0 0x7f72c0101850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:13:49.267 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:49.267+0000 7f72c6929700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f72c0101060 0x7f72c0101d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:13:49.267 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:49.267+0000 7f72c6929700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72c01022d0 con 0x7f72c01006b0 2024-05-04T12:13:49.267 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:49.267+0000 7f72c6929700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72c0102410 con 0x7f72c0101060 2024-05-04T12:13:49.267 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:49.267+0000 7f72bffff700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f72c01006b0 0x7f72c0101850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:13:49.267 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:49.267+0000 7f72bffff700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f72c01006b0 0x7f72c0101850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:51344/0 (socket says 172.21.15.79:51344) 2024-05-04T12:13:49.268 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:49.267+0000 7f72bffff700 1 -- 172.21.15.79:0/1138115150 learned_addr learned my addr 172.21.15.79:0/1138115150 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:13:49.268 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:49.267+0000 7f72bf7fe700 1 --2- 172.21.15.79:0/1138115150 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f72c0101060 0x7f72c0101d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:13:49.268 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:49.267+0000 7f72bffff700 1 -- 172.21.15.79:0/1138115150 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f72c0101060 msgr2=0x7f72c0101d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:13:49.268 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:49.267+0000 7f72bffff700 1 --2- 172.21.15.79:0/1138115150 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f72c0101060 0x7f72c0101d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:13:49.268 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:49.267+0000 7f72bffff700 1 -- 172.21.15.79:0/1138115150 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72b40095f0 con 0x7f72c01006b0 2024-05-04T12:13:49.268 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:49.267+0000 7f72bffff700 1 --2- 172.21.15.79:0/1138115150 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f72c01006b0 0x7f72c0101850 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f72b000b4b0 tx=0x7f72b000b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:13:49.268 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:49.267+0000 7f72bd7fa700 1 -- 172.21.15.79:0/1138115150 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f72b0011800 con 0x7f72c01006b0 2024-05-04T12:13:49.268 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:49.267+0000 7f72bd7fa700 1 -- 172.21.15.79:0/1138115150 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f72b0011e10 con 0x7f72c01006b0 2024-05-04T12:13:49.268 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:49.267+0000 7f72c6929700 1 -- 172.21.15.79:0/1138115150 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72c01a2ad0 con 0x7f72c01006b0 2024-05-04T12:13:49.268 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:49.267+0000 7f72bd7fa700 1 -- 172.21.15.79:0/1138115150 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f72b000f2e0 con 0x7f72c01006b0 2024-05-04T12:13:49.268 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:49.267+0000 7f72c6929700 1 -- 172.21.15.79:0/1138115150 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72c01a2ee0 con 0x7f72c01006b0 2024-05-04T12:13:49.269 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:49.269+0000 7f72c6929700 1 -- 172.21.15.79:0/1138115150 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72c0109150 con 0x7f72c01006b0 2024-05-04T12:13:49.269 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:49.269+0000 7f72bd7fa700 1 -- 172.21.15.79:0/1138115150 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f72b0010380 con 0x7f72c01006b0 2024-05-04T12:13:49.269 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:49.269+0000 7f72bd7fa700 1 --2- 172.21.15.79:0/1138115150 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f72a806c4e0 0x7f72a806e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:13:49.269 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:49.269+0000 7f72bd7fa700 1 -- 172.21.15.79:0/1138115150 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f72b008b090 con 0x7f72c01006b0 2024-05-04T12:13:49.269 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:49.269+0000 7f72bf7fe700 1 --2- 172.21.15.79:0/1138115150 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f72a806c4e0 0x7f72a806e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:13:49.269 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:49.270+0000 7f72bf7fe700 1 --2- 172.21.15.79:0/1138115150 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f72a806c4e0 0x7f72a806e9a0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f72c0103040 tx=0x7f72b401a040 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:13:49.273 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:49.274+0000 7f72bd7fa700 1 -- 172.21.15.79:0/1138115150 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f72b0055bb0 con 0x7f72c01006b0 2024-05-04T12:13:49.463 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:49.464+0000 7f72c6929700 1 -- 172.21.15.79:0/1138115150 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f72c0108b30 con 0x7f72c01006b0 2024-05-04T12:13:49.464 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:49.465+0000 7f72bd7fa700 1 -- 172.21.15.79:0/1138115150 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) v1 ==== 74+0+11347 (secure 0 0 0) 0x7f72b00591d0 con 0x7f72c01006b0 2024-05-04T12:13:49.464 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-05-04T12:13:49.465 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":40,"fsid":"d9b985c4-0a0e-11ef-bc97-c7b262605968","created":"2024-05-04T12:07:51.285061+0000","modified":"2024-05-04T12:13:44.140003+0000","last_up_change":"2024-05-04T12:13:43.138775+0000","last_in_change":"2024-05-04T12:13:23.351367+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-04T12:12:27.893460+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"904ab939-d572-43db-8c6f-fb309334b478","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6802","nonce":1309201220},{"type":"v1","addr":"172.21.15.79:6803","nonce":1309201220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6804","nonce":1309201220},{"type":"v1","addr":"172.21.15.79:6805","nonce":1309201220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6808","nonce":1309201220},{"type":"v1","addr":"172.21.15.79:6809","nonce":1309201220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6806","nonce":1309201220},{"type":"v1","addr":"172.21.15.79:6807","nonce":1309201220}]},"public_addr":"172.21.15.79:6803/1309201220","cluster_addr":"172.21.15.79:6805/1309201220","heartbeat_back_addr":"172.21.15.79:6809/1309201220","heartbeat_front_addr":"172.21.15.79:6807/1309201220","state":["exists","up"]},{"osd":1,"uuid":"bce71499-a9a8-4d8f-932e-1c85918eee61","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6810","nonce":3443379349},{"type":"v1","addr":"172.21.15.79:6811","nonce":3443379349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6812","nonce":3443379349},{"type":"v1","addr":"172.21.15.79:6813","nonce":3443379349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6816","nonce":3443379349},{"type":"v1","addr":"172.21.15.79:6817","nonce":3443379349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6814","nonce":3443379349},{"type":"v1","addr":"172.21.15.79:6815","nonce":3443379349}]},"public_addr":"172.21.15.79:6811/3443379349","cluster_addr":"172.21.15.79:6813/3443379349","heartbeat_back_addr":"172.21.15.79:6817/3443379349","heartbeat_front_addr":"172.21.15.79:6815/3443379349","state":["exists","up"]},{"osd":2,"uuid":"cfb80418-73e3-4001-8d30-53f06365d435","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6818","nonce":3776007511},{"type":"v1","addr":"172.21.15.79:6819","nonce":3776007511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6820","nonce":3776007511},{"type":"v1","addr":"172.21.15.79:6821","nonce":3776007511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6824","nonce":3776007511},{"type":"v1","addr":"172.21.15.79:6825","nonce":3776007511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6822","nonce":3776007511},{"type":"v1","addr":"172.21.15.79:6823","nonce":3776007511}]},"public_addr":"172.21.15.79:6819/3776007511","cluster_addr":"172.21.15.79:6821/3776007511","heartbeat_back_addr":"172.21.15.79:6825/3776007511","heartbeat_front_addr":"172.21.15.79:6823/3776007511","state":["exists","up"]},{"osd":3,"uuid":"67d66fd8-d092-4ba0-8418-799689ae0a04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":1187278645},{"type":"v1","addr":"172.21.15.146:6801","nonce":1187278645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":1187278645},{"type":"v1","addr":"172.21.15.146:6803","nonce":1187278645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":1187278645},{"type":"v1","addr":"172.21.15.146:6807","nonce":1187278645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":1187278645},{"type":"v1","addr":"172.21.15.146:6805","nonce":1187278645}]},"public_addr":"172.21.15.146:6801/1187278645","cluster_addr":"172.21.15.146:6803/1187278645","heartbeat_back_addr":"172.21.15.146:6807/1187278645","heartbeat_front_addr":"172.21.15.146:6805/1187278645","state":["exists","up"]},{"osd":4,"uuid":"845a6602-d767-45c8-9fd1-1cbbe89adb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":3913803877},{"type":"v1","addr":"172.21.15.146:6809","nonce":3913803877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":3913803877},{"type":"v1","addr":"172.21.15.146:6811","nonce":3913803877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":3913803877},{"type":"v1","addr":"172.21.15.146:6815","nonce":3913803877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":3913803877},{"type":"v1","addr":"172.21.15.146:6813","nonce":3913803877}]},"public_addr":"172.21.15.146:6809/3913803877","cluster_addr":"172.21.15.146:6811/3913803877","heartbeat_back_addr":"172.21.15.146:6815/3913803877","heartbeat_front_addr":"172.21.15.146:6813/3913803877","state":["exists","up"]},{"osd":5,"uuid":"02174a1e-ba23-4235-bd22-0969c319b052","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":3191460146},{"type":"v1","addr":"172.21.15.146:6817","nonce":3191460146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":3191460146},{"type":"v1","addr":"172.21.15.146:6819","nonce":3191460146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":3191460146},{"type":"v1","addr":"172.21.15.146:6823","nonce":3191460146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3191460146},{"type":"v1","addr":"172.21.15.146:6821","nonce":3191460146}]},"public_addr":"172.21.15.146:6817/3191460146","cluster_addr":"172.21.15.146:6819/3191460146","heartbeat_back_addr":"172.21.15.146:6823/3191460146","heartbeat_front_addr":"172.21.15.146:6821/3191460146","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-04T12:11:26.960942+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-04T12:11:56.001694+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-04T12:12:24.349808+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-04T12:12:48.199923+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-04T12:13:14.153595+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-04T12:13:41.185784+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.79:0/488603041":"2024-05-05T12:10:11.585392+0000","172.21.15.79:0/1526054010":"2024-05-05T12:10:11.585392+0000","172.21.15.79:6801/4263681324":"2024-05-05T12:10:11.585392+0000","172.21.15.79:6800/4263681324":"2024-05-05T12:10:11.585392+0000","172.21.15.79:0/3218172085":"2024-05-05T12:08:58.217661+0000","172.21.15.79:0/865611036":"2024-05-05T12:10:11.585392+0000","172.21.15.79:0/4109386639":"2024-05-05T12:08:58.217661+0000","172.21.15.79:0/1765481553":"2024-05-05T12:08:58.217661+0000","172.21.15.79:6801/1620705054":"2024-05-05T12:08:58.217661+0000","172.21.15.79:0/2668033337":"2024-05-05T12:08:26.031771+0000","172.21.15.79:6800/839275478":"2024-05-05T12:08:26.031771+0000","172.21.15.79:0/2735058954":"2024-05-05T12:08:26.031771+0000","172.21.15.79:0/3859814994":"2024-05-05T12:08:26.031771+0000","172.21.15.79:6801/839275478":"2024-05-05T12:08:26.031771+0000","172.21.15.79:6800/1620705054":"2024-05-05T12:08:58.217661+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-04T12:13:49.466 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:49.467+0000 7f72c6929700 1 -- 172.21.15.79:0/1138115150 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f72a806c4e0 msgr2=0x7f72a806e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:13:49.467 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:49.467+0000 7f72c6929700 1 --2- 172.21.15.79:0/1138115150 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f72a806c4e0 0x7f72a806e9a0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f72c0103040 tx=0x7f72b401a040 comp rx=0 tx=0).stop 2024-05-04T12:13:49.467 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:49.467+0000 7f72c6929700 1 -- 172.21.15.79:0/1138115150 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f72c01006b0 msgr2=0x7f72c0101850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:13:49.467 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:49.467+0000 7f72c6929700 1 --2- 172.21.15.79:0/1138115150 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f72c01006b0 0x7f72c0101850 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f72b000b4b0 tx=0x7f72b000b7c0 comp rx=0 tx=0).stop 2024-05-04T12:13:49.467 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:49.467+0000 7f72c6929700 1 -- 172.21.15.79:0/1138115150 shutdown_connections 2024-05-04T12:13:49.467 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:49.467+0000 7f72c6929700 1 --2- 172.21.15.79:0/1138115150 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f72a806c4e0 0x7f72a806e9a0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:13:49.467 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:49.467+0000 7f72c6929700 1 --2- 172.21.15.79:0/1138115150 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f72c01006b0 0x7f72c0101850 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:13:49.467 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:49.467+0000 7f72c6929700 1 --2- 172.21.15.79:0/1138115150 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f72c0101060 0x7f72c0101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:13:49.467 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:49.467+0000 7f72c6929700 1 -- 172.21.15.79:0/1138115150 >> 172.21.15.79:0/1138115150 conn(0x7f72c0074e30 msgr2=0x7f72c0103bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:13:49.467 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:49.467+0000 7f72c6929700 1 -- 172.21.15.79:0/1138115150 shutdown_connections 2024-05-04T12:13:49.467 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:49.468+0000 7f72c6929700 1 -- 172.21.15.79:0/1138115150 wait complete. 2024-05-04T12:13:49.526 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:49 smithi146 ceph-mon[35311]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-04T12:13:50.152 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-04T12:12:27.893460+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 6, 'score_stable': 6, 'optimal_score': 0.5, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-04T12:13:50.152 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid d9b985c4-0a0e-11ef-bc97-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-05-04T12:13:50.363 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/d9b985c4-0a0e-11ef-bc97-c7b262605968/mon.smithi079/config 2024-05-04T12:13:50.444 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:50 smithi079 ceph-mon[27222]: from='client.? 172.21.15.79:0/1138115150' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-04T12:13:50.526 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:50 smithi146 ceph-mon[35311]: from='client.? 172.21.15.79:0/1138115150' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-04T12:13:51.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:51 smithi079 ceph-mon[27222]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-04T12:13:51.526 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:51 smithi146 ceph-mon[35311]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-04T12:13:51.989 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:51.989+0000 7f1f3b27c700 1 -- 172.21.15.79:0/2841760346 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1f34101850 msgr2=0x7f1f341058a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:13:51.990 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:51.989+0000 7f1f3b27c700 1 --2- 172.21.15.79:0/2841760346 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1f34101850 0x7f1f341058a0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f1f24009910 tx=0x7f1f24009c20 comp rx=0 tx=0).stop 2024-05-04T12:13:51.990 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:51.989+0000 7f1f3b27c700 1 -- 172.21.15.79:0/2841760346 shutdown_connections 2024-05-04T12:13:51.990 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:51.989+0000 7f1f3b27c700 1 --2- 172.21.15.79:0/2841760346 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1f34101850 0x7f1f341058a0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:13:51.990 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:51.989+0000 7f1f3b27c700 1 --2- 172.21.15.79:0/2841760346 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1f34100ea0 0x7f1f34101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:13:51.990 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:51.989+0000 7f1f3b27c700 1 -- 172.21.15.79:0/2841760346 >> 172.21.15.79:0/2841760346 conn(0x7f1f340fc750 msgr2=0x7f1f340feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:13:51.990 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:51.989+0000 7f1f3b27c700 1 -- 172.21.15.79:0/2841760346 shutdown_connections 2024-05-04T12:13:51.990 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:51.989+0000 7f1f3b27c700 1 -- 172.21.15.79:0/2841760346 wait complete. 2024-05-04T12:13:51.990 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:51.989+0000 7f1f3b27c700 1 Processor -- start 2024-05-04T12:13:51.990 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:51.989+0000 7f1f3b27c700 1 -- start start 2024-05-04T12:13:51.990 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:51.990+0000 7f1f3b27c700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1f34100ea0 0x7f1f34073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:13:51.990 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:51.990+0000 7f1f3b27c700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1f34101850 0x7f1f340735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:13:51.991 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:51.990+0000 7f1f3b27c700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f34073b10 con 0x7f1f34101850 2024-05-04T12:13:51.991 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:51.990+0000 7f1f3b27c700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f34073c50 con 0x7f1f34100ea0 2024-05-04T12:13:51.991 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:51.990+0000 7f1f38817700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1f34101850 0x7f1f340735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:13:51.991 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:51.990+0000 7f1f38817700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1f34101850 0x7f1f340735d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:51362/0 (socket says 172.21.15.79:51362) 2024-05-04T12:13:51.991 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:51.990+0000 7f1f38817700 1 -- 172.21.15.79:0/1183611057 learned_addr learned my addr 172.21.15.79:0/1183611057 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:13:51.991 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:51.990+0000 7f1f39018700 1 --2- 172.21.15.79:0/1183611057 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1f34100ea0 0x7f1f34073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:13:51.991 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:51.990+0000 7f1f38817700 1 -- 172.21.15.79:0/1183611057 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1f34100ea0 msgr2=0x7f1f34073090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:13:51.991 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:51.990+0000 7f1f38817700 1 --2- 172.21.15.79:0/1183611057 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1f34100ea0 0x7f1f34073090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:13:51.991 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:51.990+0000 7f1f38817700 1 -- 172.21.15.79:0/1183611057 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f240095f0 con 0x7f1f34101850 2024-05-04T12:13:51.991 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:51.990+0000 7f1f39018700 1 --2- 172.21.15.79:0/1183611057 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1f34100ea0 0x7f1f34073090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-04T12:13:51.991 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:51.990+0000 7f1f38817700 1 --2- 172.21.15.79:0/1183611057 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1f34101850 0x7f1f340735d0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f1f240092e0 tx=0x7f1f2400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:13:51.991 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:51.990+0000 7f1f2a7fc700 1 -- 172.21.15.79:0/1183611057 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1f2401d070 con 0x7f1f34101850 2024-05-04T12:13:51.992 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:51.990+0000 7f1f2a7fc700 1 -- 172.21.15.79:0/1183611057 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f1f24022470 con 0x7f1f34101850 2024-05-04T12:13:51.992 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:51.990+0000 7f1f3b27c700 1 -- 172.21.15.79:0/1183611057 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f34073ed0 con 0x7f1f34101850 2024-05-04T12:13:51.992 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:51.990+0000 7f1f2a7fc700 1 -- 172.21.15.79:0/1183611057 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1f2400f690 con 0x7f1f34101850 2024-05-04T12:13:51.992 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:51.990+0000 7f1f3b27c700 1 -- 172.21.15.79:0/1183611057 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f341a2e70 con 0x7f1f34101850 2024-05-04T12:13:51.993 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:51.991+0000 7f1f3b27c700 1 -- 172.21.15.79:0/1183611057 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f341091f0 con 0x7f1f34101850 2024-05-04T12:13:51.993 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:51.992+0000 7f1f2a7fc700 1 -- 172.21.15.79:0/1183611057 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f1f240225e0 con 0x7f1f34101850 2024-05-04T12:13:51.994 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:51.992+0000 7f1f2a7fc700 1 --2- 172.21.15.79:0/1183611057 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f1f2006c4e0 0x7f1f2006e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:13:51.994 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:51.992+0000 7f1f2a7fc700 1 -- 172.21.15.79:0/1183611057 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f1f2408c9f0 con 0x7f1f34101850 2024-05-04T12:13:51.994 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:51.992+0000 7f1f39018700 1 --2- 172.21.15.79:0/1183611057 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f1f2006c4e0 0x7f1f2006e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:13:51.994 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:51.993+0000 7f1f39018700 1 --2- 172.21.15.79:0/1183611057 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f1f2006c4e0 0x7f1f2006e9a0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f1f30007900 tx=0x7f1f30008040 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:13:51.995 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:51.997+0000 7f1f2a7fc700 1 -- 172.21.15.79:0/1183611057 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f1f2408ee60 con 0x7f1f34101850 2024-05-04T12:13:52.183 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:52.184+0000 7f1f3b27c700 1 -- 172.21.15.79:0/1183611057 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f1f340745f0 con 0x7f1f34101850 2024-05-04T12:13:52.183 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:52.184+0000 7f1f2a7fc700 1 -- 172.21.15.79:0/1183611057 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v40) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1f24027020 con 0x7f1f34101850 2024-05-04T12:13:52.184 INFO:teuthology.orchestra.run.smithi079.stdout:pg_num: 1 2024-05-04T12:13:52.186 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:52.187+0000 7f1f3b27c700 1 -- 172.21.15.79:0/1183611057 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f1f2006c4e0 msgr2=0x7f1f2006e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:13:52.186 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:52.187+0000 7f1f3b27c700 1 --2- 172.21.15.79:0/1183611057 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f1f2006c4e0 0x7f1f2006e9a0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f1f30007900 tx=0x7f1f30008040 comp rx=0 tx=0).stop 2024-05-04T12:13:52.186 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:52.187+0000 7f1f3b27c700 1 -- 172.21.15.79:0/1183611057 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1f34101850 msgr2=0x7f1f340735d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:13:52.186 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:52.187+0000 7f1f3b27c700 1 --2- 172.21.15.79:0/1183611057 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1f34101850 0x7f1f340735d0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f1f240092e0 tx=0x7f1f2400be20 comp rx=0 tx=0).stop 2024-05-04T12:13:52.186 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:52.187+0000 7f1f3b27c700 1 -- 172.21.15.79:0/1183611057 shutdown_connections 2024-05-04T12:13:52.186 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:52.187+0000 7f1f3b27c700 1 --2- 172.21.15.79:0/1183611057 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f1f2006c4e0 0x7f1f2006e9a0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:13:52.186 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:52.187+0000 7f1f3b27c700 1 --2- 172.21.15.79:0/1183611057 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1f34100ea0 0x7f1f34073090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:13:52.186 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:52.187+0000 7f1f3b27c700 1 --2- 172.21.15.79:0/1183611057 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1f34101850 0x7f1f340735d0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:13:52.186 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:52.187+0000 7f1f3b27c700 1 -- 172.21.15.79:0/1183611057 >> 172.21.15.79:0/1183611057 conn(0x7f1f340fc750 msgr2=0x7f1f340fde50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:13:52.186 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:52.187+0000 7f1f3b27c700 1 -- 172.21.15.79:0/1183611057 shutdown_connections 2024-05-04T12:13:52.187 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:52.187+0000 7f1f3b27c700 1 -- 172.21.15.79:0/1183611057 wait complete. 2024-05-04T12:13:52.838 INFO:tasks.cephadm:Setting up client nodes... 2024-05-04T12:13:52.839 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d9b985c4-0a0e-11ef-bc97-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-05-04T12:13:53.043 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/d9b985c4-0a0e-11ef-bc97-c7b262605968/mon.smithi079/config 2024-05-04T12:13:53.166 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:53 smithi079 ceph-mon[27222]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-04T12:13:53.166 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:53 smithi079 ceph-mon[27222]: from='client.? 172.21.15.79:0/1183611057' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-04T12:13:53.526 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:53 smithi146 ceph-mon[35311]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-04T12:13:53.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:53 smithi146 ceph-mon[35311]: from='client.? 172.21.15.79:0/1183611057' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-04T12:13:54.570 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:54.569+0000 7ffa3e019700 1 -- 172.21.15.79:0/856450185 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ffa38101190 msgr2=0x7ffa381015b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:13:54.570 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:54.569+0000 7ffa3e019700 1 --2- 172.21.15.79:0/856450185 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ffa38101190 0x7ffa381015b0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7ffa28009910 tx=0x7ffa28009c20 comp rx=0 tx=0).stop 2024-05-04T12:13:54.570 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:54.569+0000 7ffa3e019700 1 -- 172.21.15.79:0/856450185 shutdown_connections 2024-05-04T12:13:54.570 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:54.569+0000 7ffa3e019700 1 --2- 172.21.15.79:0/856450185 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffa380ffdf0 0x7ffa38100270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:13:54.570 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:54.569+0000 7ffa3e019700 1 --2- 172.21.15.79:0/856450185 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ffa38101190 0x7ffa381015b0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:13:54.570 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:54.569+0000 7ffa3e019700 1 -- 172.21.15.79:0/856450185 >> 172.21.15.79:0/856450185 conn(0x7ffa380fa8c0 msgr2=0x7ffa380fcd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:13:54.570 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:54.569+0000 7ffa3e019700 1 -- 172.21.15.79:0/856450185 shutdown_connections 2024-05-04T12:13:54.570 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:54.569+0000 7ffa3e019700 1 -- 172.21.15.79:0/856450185 wait complete. 2024-05-04T12:13:54.570 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:54.569+0000 7ffa3e019700 1 Processor -- start 2024-05-04T12:13:54.570 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:54.569+0000 7ffa3e019700 1 -- start start 2024-05-04T12:13:54.570 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:54.570+0000 7ffa3e019700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ffa380ffdf0 0x7ffa38195440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:13:54.570 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:54.570+0000 7ffa3e019700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffa38101190 0x7ffa38195980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:13:54.571 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:54.570+0000 7ffa3e019700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa38195fa0 con 0x7ffa380ffdf0 2024-05-04T12:13:54.571 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:54.570+0000 7ffa3e019700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa381960e0 con 0x7ffa38101190 2024-05-04T12:13:54.571 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:54.570+0000 7ffa377fe700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ffa380ffdf0 0x7ffa38195440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:13:54.571 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:54.570+0000 7ffa377fe700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ffa380ffdf0 0x7ffa38195440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:41038/0 (socket says 172.21.15.79:41038) 2024-05-04T12:13:54.571 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:54.570+0000 7ffa377fe700 1 -- 172.21.15.79:0/864246863 learned_addr learned my addr 172.21.15.79:0/864246863 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:13:54.571 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:54.570+0000 7ffa2ffff700 1 --2- 172.21.15.79:0/864246863 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffa38101190 0x7ffa38195980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:13:54.571 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:54.570+0000 7ffa377fe700 1 -- 172.21.15.79:0/864246863 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffa38101190 msgr2=0x7ffa38195980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:13:54.571 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:54.570+0000 7ffa377fe700 1 --2- 172.21.15.79:0/864246863 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffa38101190 0x7ffa38195980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:13:54.571 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:54.570+0000 7ffa377fe700 1 -- 172.21.15.79:0/864246863 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa280095f0 con 0x7ffa380ffdf0 2024-05-04T12:13:54.571 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:54.570+0000 7ffa377fe700 1 --2- 172.21.15.79:0/864246863 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ffa380ffdf0 0x7ffa38195440 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7ffa280092e0 tx=0x7ffa2800bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:13:54.571 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:54.570+0000 7ffa357fa700 1 -- 172.21.15.79:0/864246863 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffa2801d070 con 0x7ffa380ffdf0 2024-05-04T12:13:54.571 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:54.570+0000 7ffa357fa700 1 -- 172.21.15.79:0/864246863 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7ffa28022470 con 0x7ffa380ffdf0 2024-05-04T12:13:54.571 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:54.570+0000 7ffa3e019700 1 -- 172.21.15.79:0/864246863 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa3819ab30 con 0x7ffa380ffdf0 2024-05-04T12:13:54.571 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:54.570+0000 7ffa357fa700 1 -- 172.21.15.79:0/864246863 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffa2800f650 con 0x7ffa380ffdf0 2024-05-04T12:13:54.572 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:54.570+0000 7ffa3e019700 1 -- 172.21.15.79:0/864246863 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa3819b020 con 0x7ffa380ffdf0 2024-05-04T12:13:54.572 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:54.571+0000 7ffa3e019700 1 -- 172.21.15.79:0/864246863 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa38108750 con 0x7ffa380ffdf0 2024-05-04T12:13:54.572 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:54.572+0000 7ffa357fa700 1 -- 172.21.15.79:0/864246863 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7ffa280225e0 con 0x7ffa380ffdf0 2024-05-04T12:13:54.573 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:54.572+0000 7ffa357fa700 1 --2- 172.21.15.79:0/864246863 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7ffa2406c4e0 0x7ffa2406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:13:54.573 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:54.572+0000 7ffa2ffff700 1 --2- 172.21.15.79:0/864246863 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7ffa2406c4e0 0x7ffa2406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:13:54.573 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:54.572+0000 7ffa357fa700 1 -- 172.21.15.79:0/864246863 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7ffa2808c9a0 con 0x7ffa380ffdf0 2024-05-04T12:13:54.573 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:54.573+0000 7ffa2ffff700 1 --2- 172.21.15.79:0/864246863 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7ffa2406c4e0 0x7ffa2406e9a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ffa38100ec0 tx=0x7ffa2000a380 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:13:54.575 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:54.577+0000 7ffa357fa700 1 -- 172.21.15.79:0/864246863 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7ffa2805b000 con 0x7ffa380ffdf0 2024-05-04T12:13:54.844 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:54.845+0000 7ffa3e019700 1 -- 172.21.15.79:0/864246863 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7ffa3819b670 con 0x7ffa380ffdf0 2024-05-04T12:13:54.847 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:54.849+0000 7ffa357fa700 1 -- 172.21.15.79:0/864246863 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v16) v1 ==== 170+0+59 (secure 0 0 0) 0x7ffa2805ab90 con 0x7ffa380ffdf0 2024-05-04T12:13:54.848 INFO:teuthology.orchestra.run.smithi079.stdout:[client.0] 2024-05-04T12:13:54.848 INFO:teuthology.orchestra.run.smithi079.stdout: key = AQCCJjZm1gN4MhAA+3YqDGpI0tmSNxrVyAwOEg== 2024-05-04T12:13:54.850 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:54.851+0000 7ffa3e019700 1 -- 172.21.15.79:0/864246863 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7ffa2406c4e0 msgr2=0x7ffa2406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:13:54.850 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:54.851+0000 7ffa3e019700 1 --2- 172.21.15.79:0/864246863 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7ffa2406c4e0 0x7ffa2406e9a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ffa38100ec0 tx=0x7ffa2000a380 comp rx=0 tx=0).stop 2024-05-04T12:13:54.850 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:54.851+0000 7ffa3e019700 1 -- 172.21.15.79:0/864246863 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ffa380ffdf0 msgr2=0x7ffa38195440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:13:54.850 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:54.851+0000 7ffa3e019700 1 --2- 172.21.15.79:0/864246863 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ffa380ffdf0 0x7ffa38195440 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7ffa280092e0 tx=0x7ffa2800bd70 comp rx=0 tx=0).stop 2024-05-04T12:13:54.851 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:54.851+0000 7ffa3e019700 1 -- 172.21.15.79:0/864246863 shutdown_connections 2024-05-04T12:13:54.851 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:54.851+0000 7ffa3e019700 1 --2- 172.21.15.79:0/864246863 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7ffa2406c4e0 0x7ffa2406e9a0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:13:54.851 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:54.851+0000 7ffa3e019700 1 --2- 172.21.15.79:0/864246863 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ffa380ffdf0 0x7ffa38195440 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:13:54.851 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:54.851+0000 7ffa3e019700 1 --2- 172.21.15.79:0/864246863 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffa38101190 0x7ffa38195980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:13:54.851 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:54.851+0000 7ffa3e019700 1 -- 172.21.15.79:0/864246863 >> 172.21.15.79:0/864246863 conn(0x7ffa380fa8c0 msgr2=0x7ffa38104030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:13:54.851 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:54.851+0000 7ffa3e019700 1 -- 172.21.15.79:0/864246863 shutdown_connections 2024-05-04T12:13:54.851 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:54.851+0000 7ffa3e019700 1 -- 172.21.15.79:0/864246863 wait complete. 2024-05-04T12:13:55.145 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:55 smithi079 ceph-mon[27222]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-04T12:13:55.146 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:55 smithi079 ceph-mon[27222]: from='client.? 172.21.15.79:0/864246863' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-04T12:13:55.146 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:55 smithi079 ceph-mon[27222]: from='client.? 172.21.15.79:0/864246863' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-05-04T12:13:55.443 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-05-04T12:13:55.443 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-05-04T12:13:55.443 DEBUG:teuthology.orchestra.run.smithi079:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-05-04T12:13:55.478 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d9b985c4-0a0e-11ef-bc97-c7b262605968 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-05-04T12:13:55.505 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:55 smithi146 ceph-mon[35311]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-04T12:13:55.506 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:55 smithi146 ceph-mon[35311]: from='client.? 172.21.15.79:0/864246863' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-04T12:13:55.506 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:55 smithi146 ceph-mon[35311]: from='client.? 172.21.15.79:0/864246863' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-05-04T12:13:55.693 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/d9b985c4-0a0e-11ef-bc97-c7b262605968/mon.smithi146/config 2024-05-04T12:13:57.290 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:13:57.292+0000 7f745feb3700 1 -- 172.21.15.146:0/2625372709 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7458101eb0 msgr2=0x7f7458102310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:13:57.291 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:13:57.292+0000 7f745feb3700 1 --2- 172.21.15.146:0/2625372709 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7458101eb0 0x7f7458102310 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f7448009910 tx=0x7f7448009c20 comp rx=0 tx=0).stop 2024-05-04T12:13:57.291 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:13:57.292+0000 7f745feb3700 1 -- 172.21.15.146:0/2625372709 shutdown_connections 2024-05-04T12:13:57.291 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:13:57.292+0000 7f745feb3700 1 --2- 172.21.15.146:0/2625372709 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7458101eb0 0x7f7458102310 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:13:57.291 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:13:57.292+0000 7f745feb3700 1 --2- 172.21.15.146:0/2625372709 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7458100cb0 0x7f74581010d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:13:57.291 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:13:57.292+0000 7f745feb3700 1 -- 172.21.15.146:0/2625372709 >> 172.21.15.146:0/2625372709 conn(0x7f74580fc230 msgr2=0x7f74580fe690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:13:57.291 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:13:57.292+0000 7f745feb3700 1 -- 172.21.15.146:0/2625372709 shutdown_connections 2024-05-04T12:13:57.291 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:13:57.292+0000 7f745feb3700 1 -- 172.21.15.146:0/2625372709 wait complete. 2024-05-04T12:13:57.291 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:57 smithi146 ceph-mon[35311]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-04T12:13:57.292 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:57 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-04T12:13:57.293 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:13:57.292+0000 7f745feb3700 1 Processor -- start 2024-05-04T12:13:57.294 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:13:57.292+0000 7f745feb3700 1 -- start start 2024-05-04T12:13:57.294 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:13:57.292+0000 7f745feb3700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7458100cb0 0x7f74581964e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:13:57.294 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:13:57.292+0000 7f745feb3700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7458101eb0 0x7f7458196a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:13:57.294 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:13:57.292+0000 7f745feb3700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7458197040 con 0x7f7458100cb0 2024-05-04T12:13:57.294 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:13:57.292+0000 7f745feb3700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7458197180 con 0x7f7458101eb0 2024-05-04T12:13:57.294 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:13:57.293+0000 7f745d44e700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7458101eb0 0x7f7458196a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:13:57.294 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:13:57.293+0000 7f745d44e700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7458101eb0 0x7f7458196a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:51536/0 (socket says 172.21.15.146:51536) 2024-05-04T12:13:57.294 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:13:57.293+0000 7f745d44e700 1 -- 172.21.15.146:0/3004708116 learned_addr learned my addr 172.21.15.146:0/3004708116 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-05-04T12:13:57.294 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:13:57.293+0000 7f745dc4f700 1 --2- 172.21.15.146:0/3004708116 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7458100cb0 0x7f74581964e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:13:57.295 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:13:57.293+0000 7f745d44e700 1 -- 172.21.15.146:0/3004708116 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7458100cb0 msgr2=0x7f74581964e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:13:57.295 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:13:57.293+0000 7f745d44e700 1 --2- 172.21.15.146:0/3004708116 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7458100cb0 0x7f74581964e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:13:57.295 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:13:57.293+0000 7f745d44e700 1 -- 172.21.15.146:0/3004708116 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74480095f0 con 0x7f7458101eb0 2024-05-04T12:13:57.295 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:13:57.293+0000 7f745d44e700 1 --2- 172.21.15.146:0/3004708116 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7458101eb0 0x7f7458196a20 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f7448006010 tx=0x7f744800bd70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:13:57.295 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:13:57.293+0000 7f744effd700 1 -- 172.21.15.146:0/3004708116 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f744801d070 con 0x7f7458101eb0 2024-05-04T12:13:57.295 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:13:57.293+0000 7f744effd700 1 -- 172.21.15.146:0/3004708116 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f744800f460 con 0x7f7458101eb0 2024-05-04T12:13:57.295 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:13:57.293+0000 7f744effd700 1 -- 172.21.15.146:0/3004708116 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7448021640 con 0x7f7458101eb0 2024-05-04T12:13:57.295 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:13:57.293+0000 7f745feb3700 1 -- 172.21.15.146:0/3004708116 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f745819bbd0 con 0x7f7458101eb0 2024-05-04T12:13:57.295 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:13:57.293+0000 7f745feb3700 1 -- 172.21.15.146:0/3004708116 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f745819c020 con 0x7f7458101eb0 2024-05-04T12:13:57.295 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:13:57.294+0000 7f745feb3700 1 -- 172.21.15.146:0/3004708116 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7458109810 con 0x7f7458101eb0 2024-05-04T12:13:57.295 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:13:57.295+0000 7f744effd700 1 -- 172.21.15.146:0/3004708116 <== mon.1 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f744800fab0 con 0x7f7458101eb0 2024-05-04T12:13:57.295 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:13:57.295+0000 7f744effd700 1 --2- 172.21.15.146:0/3004708116 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f744406c4e0 0x7f744406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:13:57.295 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:13:57.295+0000 7f744effd700 1 -- 172.21.15.146:0/3004708116 <== mon.1 v2:172.21.15.146:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f744808c830 con 0x7f7458101eb0 2024-05-04T12:13:57.295 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:13:57.295+0000 7f745dc4f700 1 --2- 172.21.15.146:0/3004708116 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f744406c4e0 0x7f744406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:13:57.296 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:13:57.296+0000 7f745dc4f700 1 --2- 172.21.15.146:0/3004708116 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f744406c4e0 0x7f744406e9a0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f7458101d10 tx=0x7f745400a380 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:13:57.298 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:13:57.300+0000 7f744effd700 1 -- 172.21.15.146:0/3004708116 <== mon.1 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f744805ade0 con 0x7f7458101eb0 2024-05-04T12:13:57.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:57 smithi079 ceph-mon[27222]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-04T12:13:57.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:57 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-04T12:13:57.558 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:13:57.560+0000 7f745feb3700 1 -- 172.21.15.146:0/3004708116 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f745819c790 con 0x7f7458101eb0 2024-05-04T12:13:57.564 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:13:57.565+0000 7f744effd700 1 -- 172.21.15.146:0/3004708116 <== mon.1 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v17) v1 ==== 170+0+59 (secure 0 0 0) 0x7f744805a970 con 0x7f7458101eb0 2024-05-04T12:13:57.564 INFO:teuthology.orchestra.run.smithi146.stdout:[client.1] 2024-05-04T12:13:57.564 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQCFJjZmCj56IRAAeBsWITlk777ttcgDyTGgMA== 2024-05-04T12:13:57.566 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:13:57.568+0000 7f745feb3700 1 -- 172.21.15.146:0/3004708116 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f744406c4e0 msgr2=0x7f744406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:13:57.566 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:13:57.568+0000 7f745feb3700 1 --2- 172.21.15.146:0/3004708116 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f744406c4e0 0x7f744406e9a0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f7458101d10 tx=0x7f745400a380 comp rx=0 tx=0).stop 2024-05-04T12:13:57.566 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:13:57.568+0000 7f745feb3700 1 -- 172.21.15.146:0/3004708116 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7458101eb0 msgr2=0x7f7458196a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:13:57.566 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:13:57.568+0000 7f745feb3700 1 --2- 172.21.15.146:0/3004708116 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7458101eb0 0x7f7458196a20 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f7448006010 tx=0x7f744800bd70 comp rx=0 tx=0).stop 2024-05-04T12:13:57.566 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:13:57.568+0000 7f745feb3700 1 -- 172.21.15.146:0/3004708116 shutdown_connections 2024-05-04T12:13:57.566 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:13:57.568+0000 7f745feb3700 1 --2- 172.21.15.146:0/3004708116 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f744406c4e0 0x7f744406e9a0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:13:57.566 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:13:57.568+0000 7f745feb3700 1 --2- 172.21.15.146:0/3004708116 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7458100cb0 0x7f74581964e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:13:57.567 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:13:57.568+0000 7f745feb3700 1 --2- 172.21.15.146:0/3004708116 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7458101eb0 0x7f7458196a20 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:13:57.567 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:13:57.568+0000 7f745feb3700 1 -- 172.21.15.146:0/3004708116 >> 172.21.15.146:0/3004708116 conn(0x7f74580fc230 msgr2=0x7f74581050e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:13:57.567 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:13:57.568+0000 7f745feb3700 1 -- 172.21.15.146:0/3004708116 shutdown_connections 2024-05-04T12:13:57.567 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:13:57.568+0000 7f745feb3700 1 -- 172.21.15.146:0/3004708116 wait complete. 2024-05-04T12:13:58.179 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:58 smithi146 ceph-mon[35311]: from='client.? 172.21.15.146:0/3004708116' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-04T12:13:58.179 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:58 smithi146 ceph-mon[35311]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-04T12:13:58.179 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:58 smithi146 ceph-mon[35311]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-05-04T12:13:58.204 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-05-04T12:13:58.204 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-05-04T12:13:58.204 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-05-04T12:13:58.281 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-05-04T12:13:58.281 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-05-04T12:13:58.281 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid d9b985c4-0a0e-11ef-bc97-c7b262605968 -- ceph mgr dump --format=json 2024-05-04T12:13:58.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:58 smithi079 ceph-mon[27222]: from='client.? 172.21.15.146:0/3004708116' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-04T12:13:58.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:58 smithi079 ceph-mon[27222]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-04T12:13:58.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:58 smithi079 ceph-mon[27222]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-05-04T12:13:58.496 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/d9b985c4-0a0e-11ef-bc97-c7b262605968/mon.smithi079/config 2024-05-04T12:13:59.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:13:59 smithi079 ceph-mon[27222]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-04T12:13:59.526 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:13:59 smithi146 ceph-mon[35311]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-04T12:13:59.991 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:59.990+0000 7f82d6536700 1 -- 172.21.15.79:0/1693465933 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f82d0100ea0 msgr2=0x7f82d0101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:13:59.991 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:59.990+0000 7f82d6536700 1 --2- 172.21.15.79:0/1693465933 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f82d0100ea0 0x7f82d0101280 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f82c4009910 tx=0x7f82c4009c20 comp rx=0 tx=0).stop 2024-05-04T12:13:59.991 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:59.990+0000 7f82d6536700 1 -- 172.21.15.79:0/1693465933 shutdown_connections 2024-05-04T12:13:59.991 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:59.990+0000 7f82d6536700 1 --2- 172.21.15.79:0/1693465933 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f82d0101850 0x7f82d01058a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:13:59.991 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:59.990+0000 7f82d6536700 1 --2- 172.21.15.79:0/1693465933 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f82d0100ea0 0x7f82d0101280 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:13:59.991 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:59.990+0000 7f82d6536700 1 -- 172.21.15.79:0/1693465933 >> 172.21.15.79:0/1693465933 conn(0x7f82d00fc750 msgr2=0x7f82d00feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:13:59.991 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:59.990+0000 7f82d6536700 1 -- 172.21.15.79:0/1693465933 shutdown_connections 2024-05-04T12:13:59.992 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:59.990+0000 7f82d6536700 1 -- 172.21.15.79:0/1693465933 wait complete. 2024-05-04T12:13:59.992 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:59.991+0000 7f82d6536700 1 Processor -- start 2024-05-04T12:13:59.992 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:59.991+0000 7f82d6536700 1 -- start start 2024-05-04T12:13:59.992 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:59.991+0000 7f82d6536700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f82d0100ea0 0x7f82d0196990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:13:59.992 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:59.991+0000 7f82d6536700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f82d0101850 0x7f82d0196ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:13:59.992 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:59.991+0000 7f82d6536700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82d0197560 con 0x7f82d0100ea0 2024-05-04T12:13:59.992 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:59.991+0000 7f82d6536700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82d019b370 con 0x7f82d0101850 2024-05-04T12:13:59.992 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:59.991+0000 7f82cffff700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f82d0100ea0 0x7f82d0196990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:13:59.992 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:59.991+0000 7f82cffff700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f82d0100ea0 0x7f82d0196990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:41056/0 (socket says 172.21.15.79:41056) 2024-05-04T12:13:59.992 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:59.991+0000 7f82cffff700 1 -- 172.21.15.79:0/1377697298 learned_addr learned my addr 172.21.15.79:0/1377697298 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:13:59.992 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:59.992+0000 7f82cf7fe700 1 --2- 172.21.15.79:0/1377697298 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f82d0101850 0x7f82d0196ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:13:59.992 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:59.992+0000 7f82cffff700 1 -- 172.21.15.79:0/1377697298 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f82d0101850 msgr2=0x7f82d0196ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:13:59.992 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:59.992+0000 7f82cffff700 1 --2- 172.21.15.79:0/1377697298 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f82d0101850 0x7f82d0196ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:13:59.992 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:59.992+0000 7f82cffff700 1 -- 172.21.15.79:0/1377697298 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82c40095f0 con 0x7f82d0100ea0 2024-05-04T12:13:59.993 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:59.992+0000 7f82cffff700 1 --2- 172.21.15.79:0/1377697298 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f82d0100ea0 0x7f82d0196990 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f82c4006010 tx=0x7f82c400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:13:59.993 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:59.992+0000 7f82cd7fa700 1 -- 172.21.15.79:0/1377697298 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f82c401d070 con 0x7f82d0100ea0 2024-05-04T12:13:59.993 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:59.992+0000 7f82d6536700 1 -- 172.21.15.79:0/1377697298 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82d019b5f0 con 0x7f82d0100ea0 2024-05-04T12:13:59.993 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:59.992+0000 7f82cd7fa700 1 -- 172.21.15.79:0/1377697298 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f82c400f460 con 0x7f82d0100ea0 2024-05-04T12:13:59.993 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:59.992+0000 7f82cd7fa700 1 -- 172.21.15.79:0/1377697298 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f82c4021640 con 0x7f82d0100ea0 2024-05-04T12:13:59.993 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:59.992+0000 7f82d6536700 1 -- 172.21.15.79:0/1377697298 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82d019bae0 con 0x7f82d0100ea0 2024-05-04T12:13:59.994 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:59.993+0000 7f82d6536700 1 -- 172.21.15.79:0/1377697298 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82d0109240 con 0x7f82d0100ea0 2024-05-04T12:13:59.994 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:59.994+0000 7f82cd7fa700 1 -- 172.21.15.79:0/1377697298 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f82c400faf0 con 0x7f82d0100ea0 2024-05-04T12:13:59.994 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:59.994+0000 7f82cd7fa700 1 --2- 172.21.15.79:0/1377697298 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f82b806c4e0 0x7f82b806e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:13:59.994 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:59.994+0000 7f82cf7fe700 1 --2- 172.21.15.79:0/1377697298 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f82b806c4e0 0x7f82b806e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:13:59.994 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:59.994+0000 7f82cd7fa700 1 -- 172.21.15.79:0/1377697298 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f82c408c870 con 0x7f82d0100ea0 2024-05-04T12:13:59.994 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:59.995+0000 7f82cf7fe700 1 --2- 172.21.15.79:0/1377697298 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f82b806c4e0 0x7f82b806e9a0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f82c0006fd0 tx=0x7f82c0008040 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:13:59.997 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:13:59.999+0000 7f82cd7fa700 1 -- 172.21.15.79:0/1377697298 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f82c405ae20 con 0x7f82d0100ea0 2024-05-04T12:14:00.253 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:00.254+0000 7f82d6536700 1 -- 172.21.15.79:0/1377697298 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f82d004f2e0 con 0x7f82d0100ea0 2024-05-04T12:14:00.256 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:00.257+0000 7f82cd7fa700 1 -- 172.21.15.79:0/1377697298 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v19) v1 ==== 74+0+173034 (secure 0 0 0) 0x7f82c405a9b0 con 0x7f82d0100ea0 2024-05-04T12:14:00.259 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-05-04T12:14:00.260 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:00.260+0000 7f82d6536700 1 -- 172.21.15.79:0/1377697298 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f82b806c4e0 msgr2=0x7f82b806e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:14:00.260 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:00.260+0000 7f82d6536700 1 --2- 172.21.15.79:0/1377697298 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f82b806c4e0 0x7f82b806e9a0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f82c0006fd0 tx=0x7f82c0008040 comp rx=0 tx=0).stop 2024-05-04T12:14:00.260 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:00.260+0000 7f82d6536700 1 -- 172.21.15.79:0/1377697298 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f82d0100ea0 msgr2=0x7f82d0196990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:14:00.260 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:00.260+0000 7f82d6536700 1 --2- 172.21.15.79:0/1377697298 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f82d0100ea0 0x7f82d0196990 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f82c4006010 tx=0x7f82c400be20 comp rx=0 tx=0).stop 2024-05-04T12:14:00.260 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:00.260+0000 7f82d6536700 1 -- 172.21.15.79:0/1377697298 shutdown_connections 2024-05-04T12:14:00.260 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:00.260+0000 7f82d6536700 1 --2- 172.21.15.79:0/1377697298 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f82b806c4e0 0x7f82b806e9a0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:00.260 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:00.260+0000 7f82d6536700 1 --2- 172.21.15.79:0/1377697298 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f82d0100ea0 0x7f82d0196990 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:00.260 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:00.260+0000 7f82d6536700 1 --2- 172.21.15.79:0/1377697298 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f82d0101850 0x7f82d0196ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:00.260 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:00.260+0000 7f82d6536700 1 -- 172.21.15.79:0/1377697298 >> 172.21.15.79:0/1377697298 conn(0x7f82d00fc750 msgr2=0x7f82d00fddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:14:00.261 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:00.260+0000 7f82d6536700 1 -- 172.21.15.79:0/1377697298 shutdown_connections 2024-05-04T12:14:00.261 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:00.260+0000 7f82d6536700 1 -- 172.21.15.79:0/1377697298 wait complete. 2024-05-04T12:14:00.933 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":19,"active_gid":14219,"active_name":"smithi079.dgcpwl","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6800","nonce":402764217},{"type":"v1","addr":"172.21.15.79:6801","nonce":402764217}]},"active_addr":"172.21.15.79:6801/402764217","active_change":"2024-05-04T12:10:11.585500+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":14244,"name":"smithi146.eubqih","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.2","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.2","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.79:8443/","prometheus":"http://172.21.15.79:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":5,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.79:0","nonce":377955414}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.79:0","nonce":3862189310}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.79:0","nonce":3337157346}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.79:0","nonce":999446020}]}]} 2024-05-04T12:14:00.937 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-05-04T12:14:00.937 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-05-04T12:14:00.937 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid d9b985c4-0a0e-11ef-bc97-c7b262605968 -- ceph osd dump --format=json 2024-05-04T12:14:01.146 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/d9b985c4-0a0e-11ef-bc97-c7b262605968/mon.smithi079/config 2024-05-04T12:14:01.195 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:14:01 smithi079 ceph-mon[27222]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-04T12:14:01.195 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:14:01 smithi079 ceph-mon[27222]: from='client.? 172.21.15.79:0/1377697298' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-05-04T12:14:01.526 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:14:01 smithi146 ceph-mon[35311]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-04T12:14:01.526 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:14:01 smithi146 ceph-mon[35311]: from='client.? 172.21.15.79:0/1377697298' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-05-04T12:14:02.800 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:02.801+0000 7fd613fe3700 1 -- 172.21.15.79:0/2789060298 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd60c105bd0 msgr2=0x7fd60c105fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:14:02.800 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:02.801+0000 7fd613fe3700 1 --2- 172.21.15.79:0/2789060298 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd60c105bd0 0x7fd60c105fb0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fd608009910 tx=0x7fd608009c20 comp rx=0 tx=0).stop 2024-05-04T12:14:02.800 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:02.801+0000 7fd613fe3700 1 -- 172.21.15.79:0/2789060298 shutdown_connections 2024-05-04T12:14:02.800 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:02.801+0000 7fd613fe3700 1 --2- 172.21.15.79:0/2789060298 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd60c0ffc20 0x7fd60c100080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:02.800 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:02.801+0000 7fd613fe3700 1 --2- 172.21.15.79:0/2789060298 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd60c105bd0 0x7fd60c105fb0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:02.801 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:02.801+0000 7fd613fe3700 1 -- 172.21.15.79:0/2789060298 >> 172.21.15.79:0/2789060298 conn(0x7fd60c075670 msgr2=0x7fd60c075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:14:02.801 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:02.801+0000 7fd613fe3700 1 -- 172.21.15.79:0/2789060298 shutdown_connections 2024-05-04T12:14:02.801 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:02.802+0000 7fd613fe3700 1 -- 172.21.15.79:0/2789060298 wait complete. 2024-05-04T12:14:02.802 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:02.802+0000 7fd613fe3700 1 Processor -- start 2024-05-04T12:14:02.802 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:02.802+0000 7fd613fe3700 1 -- start start 2024-05-04T12:14:02.802 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:02.802+0000 7fd613fe3700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd60c0ffc20 0x7fd60c197700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:14:02.802 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:02.802+0000 7fd613fe3700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd60c105bd0 0x7fd60c197c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:14:02.802 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:02.802+0000 7fd613fe3700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd60c198320 con 0x7fd60c0ffc20 2024-05-04T12:14:02.802 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:02.802+0000 7fd613fe3700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd60c198460 con 0x7fd60c105bd0 2024-05-04T12:14:02.802 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:02.802+0000 7fd611d7f700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd60c0ffc20 0x7fd60c197700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:14:02.802 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:02.802+0000 7fd611d7f700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd60c0ffc20 0x7fd60c197700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:41080/0 (socket says 172.21.15.79:41080) 2024-05-04T12:14:02.802 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:02.802+0000 7fd611d7f700 1 -- 172.21.15.79:0/3297702015 learned_addr learned my addr 172.21.15.79:0/3297702015 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:14:02.803 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:02.803+0000 7fd61157e700 1 --2- 172.21.15.79:0/3297702015 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd60c105bd0 0x7fd60c197c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:14:02.803 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:02.803+0000 7fd611d7f700 1 -- 172.21.15.79:0/3297702015 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd60c105bd0 msgr2=0x7fd60c197c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:14:02.803 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:02.803+0000 7fd611d7f700 1 --2- 172.21.15.79:0/3297702015 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd60c105bd0 0x7fd60c197c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:02.803 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:02.803+0000 7fd611d7f700 1 -- 172.21.15.79:0/3297702015 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6080095f0 con 0x7fd60c0ffc20 2024-05-04T12:14:02.803 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:02.803+0000 7fd61157e700 1 --2- 172.21.15.79:0/3297702015 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd60c105bd0 0x7fd60c197c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-04T12:14:02.803 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:02.803+0000 7fd611d7f700 1 --2- 172.21.15.79:0/3297702015 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd60c0ffc20 0x7fd60c197700 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fd608006010 tx=0x7fd60800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:14:02.803 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:02.803+0000 7fd602ffd700 1 -- 172.21.15.79:0/3297702015 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd60801d070 con 0x7fd60c0ffc20 2024-05-04T12:14:02.803 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:02.803+0000 7fd602ffd700 1 -- 172.21.15.79:0/3297702015 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fd60800f460 con 0x7fd60c0ffc20 2024-05-04T12:14:02.803 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:02.803+0000 7fd613fe3700 1 -- 172.21.15.79:0/3297702015 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd60c1a0770 con 0x7fd60c0ffc20 2024-05-04T12:14:02.803 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:02.803+0000 7fd602ffd700 1 -- 172.21.15.79:0/3297702015 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd608021640 con 0x7fd60c0ffc20 2024-05-04T12:14:02.803 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:02.803+0000 7fd613fe3700 1 -- 172.21.15.79:0/3297702015 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd60c1a0c30 con 0x7fd60c0ffc20 2024-05-04T12:14:02.804 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:02.804+0000 7fd613fe3700 1 -- 172.21.15.79:0/3297702015 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd60c1080f0 con 0x7fd60c0ffc20 2024-05-04T12:14:02.804 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:02.804+0000 7fd602ffd700 1 -- 172.21.15.79:0/3297702015 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7fd60800faf0 con 0x7fd60c0ffc20 2024-05-04T12:14:02.804 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:02.805+0000 7fd602ffd700 1 --2- 172.21.15.79:0/3297702015 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7fd5f806c4e0 0x7fd5f806e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:14:02.804 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:02.805+0000 7fd602ffd700 1 -- 172.21.15.79:0/3297702015 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fd60808c870 con 0x7fd60c0ffc20 2024-05-04T12:14:02.804 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:02.805+0000 7fd61157e700 1 --2- 172.21.15.79:0/3297702015 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7fd5f806c4e0 0x7fd5f806e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:14:02.804 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:02.805+0000 7fd61157e700 1 --2- 172.21.15.79:0/3297702015 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7fd5f806c4e0 0x7fd5f806e9a0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fd60c069210 tx=0x7fd5fc008040 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:14:02.807 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:02.809+0000 7fd602ffd700 1 -- 172.21.15.79:0/3297702015 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fd608057390 con 0x7fd60c0ffc20 2024-05-04T12:14:03.022 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:03.023+0000 7fd613fe3700 1 -- 172.21.15.79:0/3297702015 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd60c068cd0 con 0x7fd60c0ffc20 2024-05-04T12:14:03.022 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:03.024+0000 7fd602ffd700 1 -- 172.21.15.79:0/3297702015 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) v1 ==== 74+0+11347 (secure 0 0 0) 0x7fd60805a9b0 con 0x7fd60c0ffc20 2024-05-04T12:14:03.022 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-05-04T12:14:03.023 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":40,"fsid":"d9b985c4-0a0e-11ef-bc97-c7b262605968","created":"2024-05-04T12:07:51.285061+0000","modified":"2024-05-04T12:13:44.140003+0000","last_up_change":"2024-05-04T12:13:43.138775+0000","last_in_change":"2024-05-04T12:13:23.351367+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-04T12:12:27.893460+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"904ab939-d572-43db-8c6f-fb309334b478","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6802","nonce":1309201220},{"type":"v1","addr":"172.21.15.79:6803","nonce":1309201220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6804","nonce":1309201220},{"type":"v1","addr":"172.21.15.79:6805","nonce":1309201220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6808","nonce":1309201220},{"type":"v1","addr":"172.21.15.79:6809","nonce":1309201220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6806","nonce":1309201220},{"type":"v1","addr":"172.21.15.79:6807","nonce":1309201220}]},"public_addr":"172.21.15.79:6803/1309201220","cluster_addr":"172.21.15.79:6805/1309201220","heartbeat_back_addr":"172.21.15.79:6809/1309201220","heartbeat_front_addr":"172.21.15.79:6807/1309201220","state":["exists","up"]},{"osd":1,"uuid":"bce71499-a9a8-4d8f-932e-1c85918eee61","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6810","nonce":3443379349},{"type":"v1","addr":"172.21.15.79:6811","nonce":3443379349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6812","nonce":3443379349},{"type":"v1","addr":"172.21.15.79:6813","nonce":3443379349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6816","nonce":3443379349},{"type":"v1","addr":"172.21.15.79:6817","nonce":3443379349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6814","nonce":3443379349},{"type":"v1","addr":"172.21.15.79:6815","nonce":3443379349}]},"public_addr":"172.21.15.79:6811/3443379349","cluster_addr":"172.21.15.79:6813/3443379349","heartbeat_back_addr":"172.21.15.79:6817/3443379349","heartbeat_front_addr":"172.21.15.79:6815/3443379349","state":["exists","up"]},{"osd":2,"uuid":"cfb80418-73e3-4001-8d30-53f06365d435","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6818","nonce":3776007511},{"type":"v1","addr":"172.21.15.79:6819","nonce":3776007511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6820","nonce":3776007511},{"type":"v1","addr":"172.21.15.79:6821","nonce":3776007511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6824","nonce":3776007511},{"type":"v1","addr":"172.21.15.79:6825","nonce":3776007511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6822","nonce":3776007511},{"type":"v1","addr":"172.21.15.79:6823","nonce":3776007511}]},"public_addr":"172.21.15.79:6819/3776007511","cluster_addr":"172.21.15.79:6821/3776007511","heartbeat_back_addr":"172.21.15.79:6825/3776007511","heartbeat_front_addr":"172.21.15.79:6823/3776007511","state":["exists","up"]},{"osd":3,"uuid":"67d66fd8-d092-4ba0-8418-799689ae0a04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":1187278645},{"type":"v1","addr":"172.21.15.146:6801","nonce":1187278645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":1187278645},{"type":"v1","addr":"172.21.15.146:6803","nonce":1187278645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":1187278645},{"type":"v1","addr":"172.21.15.146:6807","nonce":1187278645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":1187278645},{"type":"v1","addr":"172.21.15.146:6805","nonce":1187278645}]},"public_addr":"172.21.15.146:6801/1187278645","cluster_addr":"172.21.15.146:6803/1187278645","heartbeat_back_addr":"172.21.15.146:6807/1187278645","heartbeat_front_addr":"172.21.15.146:6805/1187278645","state":["exists","up"]},{"osd":4,"uuid":"845a6602-d767-45c8-9fd1-1cbbe89adb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":3913803877},{"type":"v1","addr":"172.21.15.146:6809","nonce":3913803877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":3913803877},{"type":"v1","addr":"172.21.15.146:6811","nonce":3913803877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":3913803877},{"type":"v1","addr":"172.21.15.146:6815","nonce":3913803877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":3913803877},{"type":"v1","addr":"172.21.15.146:6813","nonce":3913803877}]},"public_addr":"172.21.15.146:6809/3913803877","cluster_addr":"172.21.15.146:6811/3913803877","heartbeat_back_addr":"172.21.15.146:6815/3913803877","heartbeat_front_addr":"172.21.15.146:6813/3913803877","state":["exists","up"]},{"osd":5,"uuid":"02174a1e-ba23-4235-bd22-0969c319b052","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":3191460146},{"type":"v1","addr":"172.21.15.146:6817","nonce":3191460146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":3191460146},{"type":"v1","addr":"172.21.15.146:6819","nonce":3191460146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":3191460146},{"type":"v1","addr":"172.21.15.146:6823","nonce":3191460146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3191460146},{"type":"v1","addr":"172.21.15.146:6821","nonce":3191460146}]},"public_addr":"172.21.15.146:6817/3191460146","cluster_addr":"172.21.15.146:6819/3191460146","heartbeat_back_addr":"172.21.15.146:6823/3191460146","heartbeat_front_addr":"172.21.15.146:6821/3191460146","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-04T12:11:26.960942+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-04T12:11:56.001694+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-04T12:12:24.349808+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-04T12:12:48.199923+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-04T12:13:14.153595+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-04T12:13:41.185784+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.79:0/488603041":"2024-05-05T12:10:11.585392+0000","172.21.15.79:0/1526054010":"2024-05-05T12:10:11.585392+0000","172.21.15.79:6801/4263681324":"2024-05-05T12:10:11.585392+0000","172.21.15.79:6800/4263681324":"2024-05-05T12:10:11.585392+0000","172.21.15.79:0/3218172085":"2024-05-05T12:08:58.217661+0000","172.21.15.79:0/865611036":"2024-05-05T12:10:11.585392+0000","172.21.15.79:0/4109386639":"2024-05-05T12:08:58.217661+0000","172.21.15.79:0/1765481553":"2024-05-05T12:08:58.217661+0000","172.21.15.79:6801/1620705054":"2024-05-05T12:08:58.217661+0000","172.21.15.79:0/2668033337":"2024-05-05T12:08:26.031771+0000","172.21.15.79:6800/839275478":"2024-05-05T12:08:26.031771+0000","172.21.15.79:0/2735058954":"2024-05-05T12:08:26.031771+0000","172.21.15.79:0/3859814994":"2024-05-05T12:08:26.031771+0000","172.21.15.79:6801/839275478":"2024-05-05T12:08:26.031771+0000","172.21.15.79:6800/1620705054":"2024-05-05T12:08:58.217661+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-04T12:14:03.024 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:03.026+0000 7fd613fe3700 1 -- 172.21.15.79:0/3297702015 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7fd5f806c4e0 msgr2=0x7fd5f806e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:14:03.024 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:03.026+0000 7fd613fe3700 1 --2- 172.21.15.79:0/3297702015 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7fd5f806c4e0 0x7fd5f806e9a0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fd60c069210 tx=0x7fd5fc008040 comp rx=0 tx=0).stop 2024-05-04T12:14:03.025 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:03.026+0000 7fd613fe3700 1 -- 172.21.15.79:0/3297702015 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd60c0ffc20 msgr2=0x7fd60c197700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:14:03.025 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:03.026+0000 7fd613fe3700 1 --2- 172.21.15.79:0/3297702015 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd60c0ffc20 0x7fd60c197700 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fd608006010 tx=0x7fd60800be20 comp rx=0 tx=0).stop 2024-05-04T12:14:03.025 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:03.026+0000 7fd613fe3700 1 -- 172.21.15.79:0/3297702015 shutdown_connections 2024-05-04T12:14:03.025 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:03.026+0000 7fd613fe3700 1 --2- 172.21.15.79:0/3297702015 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7fd5f806c4e0 0x7fd5f806e9a0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:03.025 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:03.026+0000 7fd613fe3700 1 --2- 172.21.15.79:0/3297702015 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd60c0ffc20 0x7fd60c197700 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:03.025 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:03.026+0000 7fd613fe3700 1 --2- 172.21.15.79:0/3297702015 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd60c105bd0 0x7fd60c197c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:03.025 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:03.026+0000 7fd613fe3700 1 -- 172.21.15.79:0/3297702015 >> 172.21.15.79:0/3297702015 conn(0x7fd60c075670 msgr2=0x7fd60c0fec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:14:03.025 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:03.026+0000 7fd613fe3700 1 -- 172.21.15.79:0/3297702015 shutdown_connections 2024-05-04T12:14:03.025 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:03.026+0000 7fd613fe3700 1 -- 172.21.15.79:0/3297702015 wait complete. 2024-05-04T12:14:03.374 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:14:03 smithi079 ceph-mon[27222]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-04T12:14:03.374 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:14:03 smithi079 ceph-mon[27222]: from='client.? 172.21.15.79:0/3297702015' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-04T12:14:03.526 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:14:03 smithi146 ceph-mon[35311]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-04T12:14:03.526 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:14:03 smithi146 ceph-mon[35311]: from='client.? 172.21.15.79:0/3297702015' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-04T12:14:03.672 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-05-04T12:14:03.672 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid d9b985c4-0a0e-11ef-bc97-c7b262605968 -- ceph osd dump --format=json 2024-05-04T12:14:03.892 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/d9b985c4-0a0e-11ef-bc97-c7b262605968/mon.smithi079/config 2024-05-04T12:14:05.295 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:14:05 smithi079 ceph-mon[27222]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-04T12:14:05.526 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:05.526+0000 7ff9d7597700 1 -- 172.21.15.79:0/3550605507 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff9d00ffda0 msgr2=0x7ff9d0100180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:14:05.526 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:05.526+0000 7ff9d7597700 1 --2- 172.21.15.79:0/3550605507 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff9d00ffda0 0x7ff9d0100180 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7ff9c0009910 tx=0x7ff9c0009c20 comp rx=0 tx=0).stop 2024-05-04T12:14:05.526 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:05.526+0000 7ff9d7597700 1 -- 172.21.15.79:0/3550605507 shutdown_connections 2024-05-04T12:14:05.526 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:05.526+0000 7ff9d7597700 1 --2- 172.21.15.79:0/3550605507 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff9d0100750 0x7ff9d01047a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:05.526 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:05.526+0000 7ff9d7597700 1 --2- 172.21.15.79:0/3550605507 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff9d00ffda0 0x7ff9d0100180 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:05.526 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:05.526+0000 7ff9d7597700 1 -- 172.21.15.79:0/3550605507 >> 172.21.15.79:0/3550605507 conn(0x7ff9d00fb650 msgr2=0x7ff9d00fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:14:05.526 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:05.526+0000 7ff9d7597700 1 -- 172.21.15.79:0/3550605507 shutdown_connections 2024-05-04T12:14:05.526 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:05.526+0000 7ff9d7597700 1 -- 172.21.15.79:0/3550605507 wait complete. 2024-05-04T12:14:05.526 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:05.527+0000 7ff9d7597700 1 Processor -- start 2024-05-04T12:14:05.526 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:05.527+0000 7ff9d7597700 1 -- start start 2024-05-04T12:14:05.526 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:05.527+0000 7ff9d7597700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff9d00ffda0 0x7ff9d0101840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:14:05.526 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:05.527+0000 7ff9d7597700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff9d0100750 0x7ff9d0101d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:14:05.527 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:05.527+0000 7ff9d7597700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9d01022c0 con 0x7ff9d0100750 2024-05-04T12:14:05.527 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:05.527+0000 7ff9d7597700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9d0102400 con 0x7ff9d00ffda0 2024-05-04T12:14:05.527 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:05.527+0000 7ff9d4b32700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff9d0100750 0x7ff9d0101d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:14:05.527 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:05.527+0000 7ff9d4b32700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff9d0100750 0x7ff9d0101d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:52986/0 (socket says 172.21.15.79:52986) 2024-05-04T12:14:05.527 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:05.527+0000 7ff9d4b32700 1 -- 172.21.15.79:0/3897590594 learned_addr learned my addr 172.21.15.79:0/3897590594 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:14:05.527 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:05.527+0000 7ff9d5333700 1 --2- 172.21.15.79:0/3897590594 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff9d00ffda0 0x7ff9d0101840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:14:05.527 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:05.527+0000 7ff9d4b32700 1 -- 172.21.15.79:0/3897590594 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff9d00ffda0 msgr2=0x7ff9d0101840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:14:05.527 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:05.527+0000 7ff9d4b32700 1 --2- 172.21.15.79:0/3897590594 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff9d00ffda0 0x7ff9d0101840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:05.527 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:05.527+0000 7ff9d4b32700 1 -- 172.21.15.79:0/3897590594 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9c00095f0 con 0x7ff9d0100750 2024-05-04T12:14:05.527 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:05.527+0000 7ff9d4b32700 1 --2- 172.21.15.79:0/3897590594 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff9d0100750 0x7ff9d0101d80 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7ff9cc00b480 tx=0x7ff9cc00b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:14:05.527 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:05.528+0000 7ff9c67fc700 1 -- 172.21.15.79:0/3897590594 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff9cc0118c0 con 0x7ff9d0100750 2024-05-04T12:14:05.527 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:05.528+0000 7ff9c67fc700 1 -- 172.21.15.79:0/3897590594 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7ff9cc010380 con 0x7ff9d0100750 2024-05-04T12:14:05.527 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:05.528+0000 7ff9d7597700 1 -- 172.21.15.79:0/3897590594 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9d01a2ad0 con 0x7ff9d0100750 2024-05-04T12:14:05.528 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:05.528+0000 7ff9c67fc700 1 -- 172.21.15.79:0/3897590594 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff9cc00f4f0 con 0x7ff9d0100750 2024-05-04T12:14:05.528 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:05.528+0000 7ff9d7597700 1 -- 172.21.15.79:0/3897590594 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9d01a2ee0 con 0x7ff9d0100750 2024-05-04T12:14:05.528 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:14:05 smithi146 ceph-mon[35311]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-04T12:14:05.529 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:05.529+0000 7ff9d7597700 1 -- 172.21.15.79:0/3897590594 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9d004f2e0 con 0x7ff9d0100750 2024-05-04T12:14:05.529 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:05.529+0000 7ff9c67fc700 1 -- 172.21.15.79:0/3897590594 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7ff9cc0104f0 con 0x7ff9d0100750 2024-05-04T12:14:05.529 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:05.530+0000 7ff9c67fc700 1 --2- 172.21.15.79:0/3897590594 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7ff9bc06c4e0 0x7ff9bc06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:14:05.529 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:05.530+0000 7ff9d5333700 1 --2- 172.21.15.79:0/3897590594 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7ff9bc06c4e0 0x7ff9bc06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:14:05.529 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:05.530+0000 7ff9c67fc700 1 -- 172.21.15.79:0/3897590594 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7ff9cc08b130 con 0x7ff9d0100750 2024-05-04T12:14:05.529 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:05.530+0000 7ff9d5333700 1 --2- 172.21.15.79:0/3897590594 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7ff9bc06c4e0 0x7ff9bc06e9a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ff9c000b5c0 tx=0x7ff9c001a040 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:14:05.532 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:05.534+0000 7ff9c67fc700 1 -- 172.21.15.79:0/3897590594 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7ff9cc055c50 con 0x7ff9d0100750 2024-05-04T12:14:05.722 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:05.723+0000 7ff9d7597700 1 -- 172.21.15.79:0/3897590594 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9d01a3130 con 0x7ff9d0100750 2024-05-04T12:14:05.722 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:05.724+0000 7ff9c67fc700 1 -- 172.21.15.79:0/3897590594 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) v1 ==== 74+0+11347 (secure 0 0 0) 0x7ff9cc059270 con 0x7ff9d0100750 2024-05-04T12:14:05.722 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-05-04T12:14:05.722 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":40,"fsid":"d9b985c4-0a0e-11ef-bc97-c7b262605968","created":"2024-05-04T12:07:51.285061+0000","modified":"2024-05-04T12:13:44.140003+0000","last_up_change":"2024-05-04T12:13:43.138775+0000","last_in_change":"2024-05-04T12:13:23.351367+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-04T12:12:27.893460+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"904ab939-d572-43db-8c6f-fb309334b478","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6802","nonce":1309201220},{"type":"v1","addr":"172.21.15.79:6803","nonce":1309201220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6804","nonce":1309201220},{"type":"v1","addr":"172.21.15.79:6805","nonce":1309201220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6808","nonce":1309201220},{"type":"v1","addr":"172.21.15.79:6809","nonce":1309201220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6806","nonce":1309201220},{"type":"v1","addr":"172.21.15.79:6807","nonce":1309201220}]},"public_addr":"172.21.15.79:6803/1309201220","cluster_addr":"172.21.15.79:6805/1309201220","heartbeat_back_addr":"172.21.15.79:6809/1309201220","heartbeat_front_addr":"172.21.15.79:6807/1309201220","state":["exists","up"]},{"osd":1,"uuid":"bce71499-a9a8-4d8f-932e-1c85918eee61","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6810","nonce":3443379349},{"type":"v1","addr":"172.21.15.79:6811","nonce":3443379349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6812","nonce":3443379349},{"type":"v1","addr":"172.21.15.79:6813","nonce":3443379349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6816","nonce":3443379349},{"type":"v1","addr":"172.21.15.79:6817","nonce":3443379349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6814","nonce":3443379349},{"type":"v1","addr":"172.21.15.79:6815","nonce":3443379349}]},"public_addr":"172.21.15.79:6811/3443379349","cluster_addr":"172.21.15.79:6813/3443379349","heartbeat_back_addr":"172.21.15.79:6817/3443379349","heartbeat_front_addr":"172.21.15.79:6815/3443379349","state":["exists","up"]},{"osd":2,"uuid":"cfb80418-73e3-4001-8d30-53f06365d435","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6818","nonce":3776007511},{"type":"v1","addr":"172.21.15.79:6819","nonce":3776007511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6820","nonce":3776007511},{"type":"v1","addr":"172.21.15.79:6821","nonce":3776007511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6824","nonce":3776007511},{"type":"v1","addr":"172.21.15.79:6825","nonce":3776007511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6822","nonce":3776007511},{"type":"v1","addr":"172.21.15.79:6823","nonce":3776007511}]},"public_addr":"172.21.15.79:6819/3776007511","cluster_addr":"172.21.15.79:6821/3776007511","heartbeat_back_addr":"172.21.15.79:6825/3776007511","heartbeat_front_addr":"172.21.15.79:6823/3776007511","state":["exists","up"]},{"osd":3,"uuid":"67d66fd8-d092-4ba0-8418-799689ae0a04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":1187278645},{"type":"v1","addr":"172.21.15.146:6801","nonce":1187278645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":1187278645},{"type":"v1","addr":"172.21.15.146:6803","nonce":1187278645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":1187278645},{"type":"v1","addr":"172.21.15.146:6807","nonce":1187278645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":1187278645},{"type":"v1","addr":"172.21.15.146:6805","nonce":1187278645}]},"public_addr":"172.21.15.146:6801/1187278645","cluster_addr":"172.21.15.146:6803/1187278645","heartbeat_back_addr":"172.21.15.146:6807/1187278645","heartbeat_front_addr":"172.21.15.146:6805/1187278645","state":["exists","up"]},{"osd":4,"uuid":"845a6602-d767-45c8-9fd1-1cbbe89adb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":3913803877},{"type":"v1","addr":"172.21.15.146:6809","nonce":3913803877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":3913803877},{"type":"v1","addr":"172.21.15.146:6811","nonce":3913803877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":3913803877},{"type":"v1","addr":"172.21.15.146:6815","nonce":3913803877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":3913803877},{"type":"v1","addr":"172.21.15.146:6813","nonce":3913803877}]},"public_addr":"172.21.15.146:6809/3913803877","cluster_addr":"172.21.15.146:6811/3913803877","heartbeat_back_addr":"172.21.15.146:6815/3913803877","heartbeat_front_addr":"172.21.15.146:6813/3913803877","state":["exists","up"]},{"osd":5,"uuid":"02174a1e-ba23-4235-bd22-0969c319b052","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":3191460146},{"type":"v1","addr":"172.21.15.146:6817","nonce":3191460146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":3191460146},{"type":"v1","addr":"172.21.15.146:6819","nonce":3191460146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":3191460146},{"type":"v1","addr":"172.21.15.146:6823","nonce":3191460146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3191460146},{"type":"v1","addr":"172.21.15.146:6821","nonce":3191460146}]},"public_addr":"172.21.15.146:6817/3191460146","cluster_addr":"172.21.15.146:6819/3191460146","heartbeat_back_addr":"172.21.15.146:6823/3191460146","heartbeat_front_addr":"172.21.15.146:6821/3191460146","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-04T12:11:26.960942+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-04T12:11:56.001694+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-04T12:12:24.349808+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-04T12:12:48.199923+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-04T12:13:14.153595+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-04T12:13:41.185784+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.79:0/488603041":"2024-05-05T12:10:11.585392+0000","172.21.15.79:0/1526054010":"2024-05-05T12:10:11.585392+0000","172.21.15.79:6801/4263681324":"2024-05-05T12:10:11.585392+0000","172.21.15.79:6800/4263681324":"2024-05-05T12:10:11.585392+0000","172.21.15.79:0/3218172085":"2024-05-05T12:08:58.217661+0000","172.21.15.79:0/865611036":"2024-05-05T12:10:11.585392+0000","172.21.15.79:0/4109386639":"2024-05-05T12:08:58.217661+0000","172.21.15.79:0/1765481553":"2024-05-05T12:08:58.217661+0000","172.21.15.79:6801/1620705054":"2024-05-05T12:08:58.217661+0000","172.21.15.79:0/2668033337":"2024-05-05T12:08:26.031771+0000","172.21.15.79:6800/839275478":"2024-05-05T12:08:26.031771+0000","172.21.15.79:0/2735058954":"2024-05-05T12:08:26.031771+0000","172.21.15.79:0/3859814994":"2024-05-05T12:08:26.031771+0000","172.21.15.79:6801/839275478":"2024-05-05T12:08:26.031771+0000","172.21.15.79:6800/1620705054":"2024-05-05T12:08:58.217661+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-04T12:14:05.724 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:05.726+0000 7ff9d7597700 1 -- 172.21.15.79:0/3897590594 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7ff9bc06c4e0 msgr2=0x7ff9bc06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:14:05.724 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:05.726+0000 7ff9d7597700 1 --2- 172.21.15.79:0/3897590594 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7ff9bc06c4e0 0x7ff9bc06e9a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ff9c000b5c0 tx=0x7ff9c001a040 comp rx=0 tx=0).stop 2024-05-04T12:14:05.725 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:05.726+0000 7ff9d7597700 1 -- 172.21.15.79:0/3897590594 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff9d0100750 msgr2=0x7ff9d0101d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:14:05.725 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:05.726+0000 7ff9d7597700 1 --2- 172.21.15.79:0/3897590594 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff9d0100750 0x7ff9d0101d80 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7ff9cc00b480 tx=0x7ff9cc00b840 comp rx=0 tx=0).stop 2024-05-04T12:14:05.725 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:05.726+0000 7ff9d7597700 1 -- 172.21.15.79:0/3897590594 shutdown_connections 2024-05-04T12:14:05.725 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:05.726+0000 7ff9d7597700 1 --2- 172.21.15.79:0/3897590594 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7ff9bc06c4e0 0x7ff9bc06e9a0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:05.725 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:05.726+0000 7ff9d7597700 1 --2- 172.21.15.79:0/3897590594 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff9d00ffda0 0x7ff9d0101840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:05.725 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:05.726+0000 7ff9d7597700 1 --2- 172.21.15.79:0/3897590594 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff9d0100750 0x7ff9d0101d80 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:05.725 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:05.726+0000 7ff9d7597700 1 -- 172.21.15.79:0/3897590594 >> 172.21.15.79:0/3897590594 conn(0x7ff9d00fb650 msgr2=0x7ff9d0103f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:14:05.725 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:05.726+0000 7ff9d7597700 1 -- 172.21.15.79:0/3897590594 shutdown_connections 2024-05-04T12:14:05.725 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:05.726+0000 7ff9d7597700 1 -- 172.21.15.79:0/3897590594 wait complete. 2024-05-04T12:14:06.430 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:14:06 smithi079 ceph-mon[27222]: from='client.? 172.21.15.79:0/3897590594' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-04T12:14:06.452 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid d9b985c4-0a0e-11ef-bc97-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-05-04T12:14:06.453 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid d9b985c4-0a0e-11ef-bc97-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-05-04T12:14:06.453 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid d9b985c4-0a0e-11ef-bc97-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-05-04T12:14:06.453 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid d9b985c4-0a0e-11ef-bc97-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-05-04T12:14:06.454 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid d9b985c4-0a0e-11ef-bc97-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-05-04T12:14:06.454 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid d9b985c4-0a0e-11ef-bc97-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-05-04T12:14:06.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:14:06 smithi146 ceph-mon[35311]: from='client.? 172.21.15.79:0/3897590594' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-04T12:14:06.743 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/d9b985c4-0a0e-11ef-bc97-c7b262605968/mon.smithi079/config 2024-05-04T12:14:06.744 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/d9b985c4-0a0e-11ef-bc97-c7b262605968/mon.smithi079/config 2024-05-04T12:14:06.807 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/d9b985c4-0a0e-11ef-bc97-c7b262605968/mon.smithi079/config 2024-05-04T12:14:06.813 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/d9b985c4-0a0e-11ef-bc97-c7b262605968/mon.smithi079/config 2024-05-04T12:14:06.813 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/d9b985c4-0a0e-11ef-bc97-c7b262605968/mon.smithi079/config 2024-05-04T12:14:07.195 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:14:07 smithi079 ceph-mon[27222]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-04T12:14:07.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:14:07 smithi146 ceph-mon[35311]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-04T12:14:08.439 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/d9b985c4-0a0e-11ef-bc97-c7b262605968/mon.smithi079/config 2024-05-04T12:14:09.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:14:09 smithi079 ceph-mon[27222]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-04T12:14:09.526 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:14:09 smithi146 ceph-mon[35311]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-04T12:14:11.240 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:14:11 smithi079 ceph-mon[27222]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-04T12:14:11.342 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:14:11 smithi146 ceph-mon[35311]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-04T12:14:11.820 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:11.819+0000 7f866adc6700 1 -- 172.21.15.79:0/3534947967 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8664100ea0 msgr2=0x7f8664101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:14:11.820 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:11.819+0000 7f866adc6700 1 --2- 172.21.15.79:0/3534947967 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8664100ea0 0x7f8664101280 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f8658009910 tx=0x7f8658009c20 comp rx=0 tx=0).stop 2024-05-04T12:14:11.820 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:11.820+0000 7f866adc6700 1 -- 172.21.15.79:0/3534947967 shutdown_connections 2024-05-04T12:14:11.820 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:11.820+0000 7f866adc6700 1 --2- 172.21.15.79:0/3534947967 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8664101850 0x7f86641058a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:11.821 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:11.820+0000 7f866adc6700 1 --2- 172.21.15.79:0/3534947967 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8664100ea0 0x7f8664101280 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:11.821 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:11.820+0000 7f866adc6700 1 -- 172.21.15.79:0/3534947967 >> 172.21.15.79:0/3534947967 conn(0x7f86640fc750 msgr2=0x7f86640feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:14:11.821 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:11.820+0000 7f866adc6700 1 -- 172.21.15.79:0/3534947967 shutdown_connections 2024-05-04T12:14:11.821 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:11.820+0000 7f866adc6700 1 -- 172.21.15.79:0/3534947967 wait complete. 2024-05-04T12:14:11.821 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:11.820+0000 7f866adc6700 1 Processor -- start 2024-05-04T12:14:11.821 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:11.820+0000 7f866adc6700 1 -- start start 2024-05-04T12:14:11.821 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:11.820+0000 7f866adc6700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8664100ea0 0x7f866419c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:14:11.821 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:11.820+0000 7f866adc6700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8664101850 0x7f866419cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:14:11.821 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:11.820+0000 7f866adc6700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f866419d2c0 con 0x7f8664101850 2024-05-04T12:14:11.821 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:11.820+0000 7f866adc6700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8664196770 con 0x7f8664100ea0 2024-05-04T12:14:11.821 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:11.821+0000 7f8663fff700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8664101850 0x7f866419cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:14:11.821 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:11.821+0000 7f8663fff700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8664101850 0x7f866419cc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:53010/0 (socket says 172.21.15.79:53010) 2024-05-04T12:14:11.821 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:11.821+0000 7f8663fff700 1 -- 172.21.15.79:0/2225871187 learned_addr learned my addr 172.21.15.79:0/2225871187 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:14:11.821 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:11.821+0000 7f8668b62700 1 --2- 172.21.15.79:0/2225871187 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8664100ea0 0x7f866419c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:14:11.821 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:11.821+0000 7f8663fff700 1 -- 172.21.15.79:0/2225871187 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8664100ea0 msgr2=0x7f866419c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:14:11.822 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:11.821+0000 7f8663fff700 1 --2- 172.21.15.79:0/2225871187 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8664100ea0 0x7f866419c6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:11.822 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:11.821+0000 7f8663fff700 1 -- 172.21.15.79:0/2225871187 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86580095f0 con 0x7f8664101850 2024-05-04T12:14:11.822 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:11.821+0000 7f8663fff700 1 --2- 172.21.15.79:0/2225871187 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8664101850 0x7f866419cc30 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f865000d8d0 tx=0x7f865000dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:14:11.822 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:11.821+0000 7f8661ffb700 1 -- 172.21.15.79:0/2225871187 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f86500041f0 con 0x7f8664101850 2024-05-04T12:14:11.822 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:11.821+0000 7f8661ffb700 1 -- 172.21.15.79:0/2225871187 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f8650007620 con 0x7f8664101850 2024-05-04T12:14:11.822 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:11.821+0000 7f866adc6700 1 -- 172.21.15.79:0/2225871187 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8664196a50 con 0x7f8664101850 2024-05-04T12:14:11.822 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:11.821+0000 7f8661ffb700 1 -- 172.21.15.79:0/2225871187 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f86500056b0 con 0x7f8664101850 2024-05-04T12:14:11.822 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:11.821+0000 7f866adc6700 1 -- 172.21.15.79:0/2225871187 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8664196fa0 con 0x7f8664101850 2024-05-04T12:14:11.823 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:11.822+0000 7f866adc6700 1 -- 172.21.15.79:0/2225871187 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f866410e0b0 con 0x7f8664101850 2024-05-04T12:14:11.823 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:11.822+0000 7f8661ffb700 1 -- 172.21.15.79:0/2225871187 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f8650004750 con 0x7f8664101850 2024-05-04T12:14:11.823 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:11.823+0000 7f8661ffb700 1 --2- 172.21.15.79:0/2225871187 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f865406c4e0 0x7f865406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:14:11.824 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:11.823+0000 7f8661ffb700 1 -- 172.21.15.79:0/2225871187 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f865008b3a0 con 0x7f8664101850 2024-05-04T12:14:11.824 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:11.823+0000 7f8668b62700 1 --2- 172.21.15.79:0/2225871187 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f865406c4e0 0x7f865406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:14:11.824 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:11.823+0000 7f8661ffb700 1 --2- 172.21.15.79:0/2225871187 >> [v2:172.21.15.146:6800/1187278645,v1:172.21.15.146:6801/1187278645] conn(0x7f86540721c0 0x7f86540745e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:14:11.824 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:11.823+0000 7f8661ffb700 1 -- 172.21.15.79:0/2225871187 --> [v2:172.21.15.146:6800/1187278645,v1:172.21.15.146:6801/1187278645] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8654074c90 con 0x7f86540721c0 2024-05-04T12:14:11.824 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:11.823+0000 7f8661ffb700 1 -- 172.21.15.79:0/2225871187 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f865008da70 con 0x7f8664101850 2024-05-04T12:14:11.824 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:11.823+0000 7f8669363700 1 --2- 172.21.15.79:0/2225871187 >> [v2:172.21.15.146:6800/1187278645,v1:172.21.15.146:6801/1187278645] conn(0x7f86540721c0 0x7f86540745e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:14:11.824 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:11.823+0000 7f8668b62700 1 --2- 172.21.15.79:0/2225871187 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f865406c4e0 0x7f865406e9a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f8658006010 tx=0x7f865801a040 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:14:11.824 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:11.824+0000 7f8669363700 1 --2- 172.21.15.79:0/2225871187 >> [v2:172.21.15.146:6800/1187278645,v1:172.21.15.146:6801/1187278645] conn(0x7f86540721c0 0x7f86540745e0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:14:11.824 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:11.824+0000 7f8661ffb700 1 -- 172.21.15.79:0/2225871187 <== osd.3 v2:172.21.15.146:6800/1187278645 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f8654074c90 con 0x7f86540721c0 2024-05-04T12:14:11.840 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:11.840+0000 7f866adc6700 1 -- 172.21.15.79:0/2225871187 --> [v2:172.21.15.146:6800/1187278645,v1:172.21.15.146:6801/1187278645] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f866404f2e0 con 0x7f86540721c0 2024-05-04T12:14:11.840 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:11.840+0000 7f8661ffb700 1 -- 172.21.15.79:0/2225871187 <== osd.3 v2:172.21.15.146:6800/1187278645 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f866404f2e0 con 0x7f86540721c0 2024-05-04T12:14:11.841 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:11.841+0000 7f866adc6700 1 -- 172.21.15.79:0/2225871187 >> [v2:172.21.15.146:6800/1187278645,v1:172.21.15.146:6801/1187278645] conn(0x7f86540721c0 msgr2=0x7f86540745e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:14:11.841 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:11.841+0000 7f866adc6700 1 --2- 172.21.15.79:0/2225871187 >> [v2:172.21.15.146:6800/1187278645,v1:172.21.15.146:6801/1187278645] conn(0x7f86540721c0 0x7f86540745e0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:11.841 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:11.841+0000 7f866adc6700 1 -- 172.21.15.79:0/2225871187 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f865406c4e0 msgr2=0x7f865406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:14:11.841 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:11.841+0000 7f866adc6700 1 --2- 172.21.15.79:0/2225871187 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f865406c4e0 0x7f865406e9a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f8658006010 tx=0x7f865801a040 comp rx=0 tx=0).stop 2024-05-04T12:14:11.841 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:11.841+0000 7f866adc6700 1 -- 172.21.15.79:0/2225871187 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8664101850 msgr2=0x7f866419cc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:14:11.841 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:11.841+0000 7f866adc6700 1 --2- 172.21.15.79:0/2225871187 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8664101850 0x7f866419cc30 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f865000d8d0 tx=0x7f865000dbe0 comp rx=0 tx=0).stop 2024-05-04T12:14:11.841 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:11.841+0000 7f866adc6700 1 -- 172.21.15.79:0/2225871187 shutdown_connections 2024-05-04T12:14:11.841 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:11.841+0000 7f866adc6700 1 --2- 172.21.15.79:0/2225871187 >> [v2:172.21.15.146:6800/1187278645,v1:172.21.15.146:6801/1187278645] conn(0x7f86540721c0 0x7f86540745e0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:11.841 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:11.841+0000 7f866adc6700 1 --2- 172.21.15.79:0/2225871187 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f865406c4e0 0x7f865406e9a0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:11.841 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:11.841+0000 7f866adc6700 1 --2- 172.21.15.79:0/2225871187 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8664100ea0 0x7f866419c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:11.841 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:11.841+0000 7f866adc6700 1 --2- 172.21.15.79:0/2225871187 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8664101850 0x7f866419cc30 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:11.841 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:11.841+0000 7f866adc6700 1 -- 172.21.15.79:0/2225871187 >> 172.21.15.79:0/2225871187 conn(0x7f86640fc750 msgr2=0x7f86640fddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:14:11.842 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:11.841+0000 7f866adc6700 1 -- 172.21.15.79:0/2225871187 shutdown_connections 2024-05-04T12:14:11.842 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:11.841+0000 7f866adc6700 1 -- 172.21.15.79:0/2225871187 wait complete. 2024-05-04T12:14:12.406 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:14:12 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-04T12:14:12.523 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:14:12 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-04T12:14:13.195 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:14:13 smithi079 ceph-mon[27222]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-04T12:14:13.526 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:14:13 smithi146 ceph-mon[35311]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-04T12:14:14.110 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:14.109+0000 7fc0da00c700 1 -- 172.21.15.79:0/306764741 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc0d4105bf0 msgr2=0x7fc0d4105fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:14:14.110 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:14.109+0000 7fc0da00c700 1 --2- 172.21.15.79:0/306764741 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc0d4105bf0 0x7fc0d4105fd0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fc0c4009910 tx=0x7fc0c4009c20 comp rx=0 tx=0).stop 2024-05-04T12:14:14.110 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:14.109+0000 7fc0da00c700 1 -- 172.21.15.79:0/306764741 shutdown_connections 2024-05-04T12:14:14.110 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:14.109+0000 7fc0da00c700 1 --2- 172.21.15.79:0/306764741 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc0d40ffb40 0x7fc0d40fffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:14.110 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:14.109+0000 7fc0da00c700 1 --2- 172.21.15.79:0/306764741 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc0d4105bf0 0x7fc0d4105fd0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:14.111 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:14.109+0000 7fc0da00c700 1 -- 172.21.15.79:0/306764741 >> 172.21.15.79:0/306764741 conn(0x7fc0d40fb610 msgr2=0x7fc0d40fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:14:14.111 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:14.109+0000 7fc0da00c700 1 -- 172.21.15.79:0/306764741 shutdown_connections 2024-05-04T12:14:14.111 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:14.109+0000 7fc0da00c700 1 -- 172.21.15.79:0/306764741 wait complete. 2024-05-04T12:14:14.111 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:14.109+0000 7fc0da00c700 1 Processor -- start 2024-05-04T12:14:14.111 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:14.110+0000 7fc0da00c700 1 -- start start 2024-05-04T12:14:14.111 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:14.110+0000 7fc0da00c700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc0d40ffb40 0x7fc0d419c660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:14:14.111 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:14.110+0000 7fc0da00c700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc0d4105bf0 0x7fc0d419cba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:14:14.111 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:14.110+0000 7fc0da00c700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0d419d230 con 0x7fc0d4105bf0 2024-05-04T12:14:14.111 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:14.110+0000 7fc0da00c700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0d41966e0 con 0x7fc0d40ffb40 2024-05-04T12:14:14.111 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:14.110+0000 7fc0cbfff700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc0d4105bf0 0x7fc0d419cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:14:14.111 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:14.110+0000 7fc0d37fe700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc0d40ffb40 0x7fc0d419c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:14:14.111 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:14.110+0000 7fc0d37fe700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc0d40ffb40 0x7fc0d419c660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.79:33696/0 (socket says 172.21.15.79:33696) 2024-05-04T12:14:14.112 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:14.110+0000 7fc0d37fe700 1 -- 172.21.15.79:0/862056159 learned_addr learned my addr 172.21.15.79:0/862056159 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:14:14.112 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:14.110+0000 7fc0cbfff700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc0d4105bf0 0x7fc0d419cba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:53458/0 (socket says 172.21.15.79:53458) 2024-05-04T12:14:14.112 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:14.110+0000 7fc0cbfff700 1 -- 172.21.15.79:0/862056159 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc0d40ffb40 msgr2=0x7fc0d419c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:14:14.112 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:14.110+0000 7fc0cbfff700 1 --2- 172.21.15.79:0/862056159 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc0d40ffb40 0x7fc0d419c660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:14.112 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:14.110+0000 7fc0cbfff700 1 -- 172.21.15.79:0/862056159 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0c40095f0 con 0x7fc0d4105bf0 2024-05-04T12:14:14.112 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:14.110+0000 7fc0cbfff700 1 --2- 172.21.15.79:0/862056159 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc0d4105bf0 0x7fc0d419cba0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fc0bc00b3d0 tx=0x7fc0bc00b6e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:14:14.112 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:14.110+0000 7fc0d17fa700 1 -- 172.21.15.79:0/862056159 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc0bc011800 con 0x7fc0d4105bf0 2024-05-04T12:14:14.112 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:14.111+0000 7fc0d17fa700 1 -- 172.21.15.79:0/862056159 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fc0bc00be90 con 0x7fc0d4105bf0 2024-05-04T12:14:14.112 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:14.111+0000 7fc0da00c700 1 -- 172.21.15.79:0/862056159 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0d41969c0 con 0x7fc0d4105bf0 2024-05-04T12:14:14.112 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:14.111+0000 7fc0d17fa700 1 -- 172.21.15.79:0/862056159 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc0bc010e70 con 0x7fc0d4105bf0 2024-05-04T12:14:14.112 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:14.111+0000 7fc0da00c700 1 -- 172.21.15.79:0/862056159 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0d4196f10 con 0x7fc0d4105bf0 2024-05-04T12:14:14.112 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:14.111+0000 7fc0da00c700 1 -- 172.21.15.79:0/862056159 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fc0d410d070 con 0x7fc0d4105bf0 2024-05-04T12:14:14.113 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:14.112+0000 7fc0d17fa700 1 -- 172.21.15.79:0/862056159 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7fc0bc01a400 con 0x7fc0d4105bf0 2024-05-04T12:14:14.114 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:14.112+0000 7fc0d17fa700 1 --2- 172.21.15.79:0/862056159 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7fc0c006c4e0 0x7fc0c006e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:14:14.114 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:14.112+0000 7fc0d37fe700 1 --2- 172.21.15.79:0/862056159 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7fc0c006c4e0 0x7fc0c006e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:14:14.114 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:14.112+0000 7fc0d17fa700 1 -- 172.21.15.79:0/862056159 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fc0bc08ae90 con 0x7fc0d4105bf0 2024-05-04T12:14:14.114 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:14.113+0000 7fc0d17fa700 1 --2- 172.21.15.79:0/862056159 >> [v2:172.21.15.146:6816/3191460146,v1:172.21.15.146:6817/3191460146] conn(0x7fc0c00721c0 0x7fc0c00745e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:14:14.114 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:14.113+0000 7fc0d17fa700 1 -- 172.21.15.79:0/862056159 --> [v2:172.21.15.146:6816/3191460146,v1:172.21.15.146:6817/3191460146] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc0c0074c90 con 0x7fc0c00721c0 2024-05-04T12:14:14.114 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:14.113+0000 7fc0d17fa700 1 -- 172.21.15.79:0/862056159 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc0bc090570 con 0x7fc0d4105bf0 2024-05-04T12:14:14.114 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:14.113+0000 7fc0d37fe700 1 --2- 172.21.15.79:0/862056159 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7fc0c006c4e0 0x7fc0c006e9a0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fc0c4009fd0 tx=0x7fc0c401a040 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:14:14.114 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:14.113+0000 7fc0d3fff700 1 --2- 172.21.15.79:0/862056159 >> [v2:172.21.15.146:6816/3191460146,v1:172.21.15.146:6817/3191460146] conn(0x7fc0c00721c0 0x7fc0c00745e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:14:14.114 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:14.113+0000 7fc0d3fff700 1 --2- 172.21.15.79:0/862056159 >> [v2:172.21.15.146:6816/3191460146,v1:172.21.15.146:6817/3191460146] conn(0x7fc0c00721c0 0x7fc0c00745e0 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:14:14.114 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:14.114+0000 7fc0d17fa700 1 -- 172.21.15.79:0/862056159 <== osd.5 v2:172.21.15.146:6816/3191460146 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fc0c0074c90 con 0x7fc0c00721c0 2024-05-04T12:14:14.128 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:14.129+0000 7fc0da00c700 1 -- 172.21.15.79:0/862056159 --> [v2:172.21.15.146:6816/3191460146,v1:172.21.15.146:6817/3191460146] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fc0d40679c0 con 0x7fc0c00721c0 2024-05-04T12:14:14.129 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:14.130+0000 7fc0d17fa700 1 -- 172.21.15.79:0/862056159 <== osd.5 v2:172.21.15.146:6816/3191460146 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fc0d40679c0 con 0x7fc0c00721c0 2024-05-04T12:14:14.130 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:14.130+0000 7fc0da00c700 1 -- 172.21.15.79:0/862056159 >> [v2:172.21.15.146:6816/3191460146,v1:172.21.15.146:6817/3191460146] conn(0x7fc0c00721c0 msgr2=0x7fc0c00745e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:14:14.130 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:14.130+0000 7fc0da00c700 1 --2- 172.21.15.79:0/862056159 >> [v2:172.21.15.146:6816/3191460146,v1:172.21.15.146:6817/3191460146] conn(0x7fc0c00721c0 0x7fc0c00745e0 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:14.130 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:14.130+0000 7fc0da00c700 1 -- 172.21.15.79:0/862056159 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7fc0c006c4e0 msgr2=0x7fc0c006e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:14:14.130 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:14.130+0000 7fc0da00c700 1 --2- 172.21.15.79:0/862056159 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7fc0c006c4e0 0x7fc0c006e9a0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fc0c4009fd0 tx=0x7fc0c401a040 comp rx=0 tx=0).stop 2024-05-04T12:14:14.130 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:14.130+0000 7fc0da00c700 1 -- 172.21.15.79:0/862056159 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc0d4105bf0 msgr2=0x7fc0d419cba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:14:14.130 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:14.130+0000 7fc0da00c700 1 --2- 172.21.15.79:0/862056159 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc0d4105bf0 0x7fc0d419cba0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fc0bc00b3d0 tx=0x7fc0bc00b6e0 comp rx=0 tx=0).stop 2024-05-04T12:14:14.130 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:14.130+0000 7fc0da00c700 1 -- 172.21.15.79:0/862056159 shutdown_connections 2024-05-04T12:14:14.130 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:14.130+0000 7fc0da00c700 1 --2- 172.21.15.79:0/862056159 >> [v2:172.21.15.146:6816/3191460146,v1:172.21.15.146:6817/3191460146] conn(0x7fc0c00721c0 0x7fc0c00745e0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:14.130 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:14.130+0000 7fc0da00c700 1 --2- 172.21.15.79:0/862056159 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7fc0c006c4e0 0x7fc0c006e9a0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:14.130 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:14.130+0000 7fc0da00c700 1 --2- 172.21.15.79:0/862056159 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc0d40ffb40 0x7fc0d419c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:14.130 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:14.130+0000 7fc0da00c700 1 --2- 172.21.15.79:0/862056159 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc0d4105bf0 0x7fc0d419cba0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:14.130 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:14.130+0000 7fc0da00c700 1 -- 172.21.15.79:0/862056159 >> 172.21.15.79:0/862056159 conn(0x7fc0d40fb610 msgr2=0x7fc0d4068f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:14:14.130 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:14.130+0000 7fc0da00c700 1 -- 172.21.15.79:0/862056159 shutdown_connections 2024-05-04T12:14:14.130 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:14.130+0000 7fc0da00c700 1 -- 172.21.15.79:0/862056159 wait complete. 2024-05-04T12:14:15.195 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:14:15 smithi079 ceph-mon[27222]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-04T12:14:15.460 INFO:teuthology.orchestra.run.smithi079.stdout:111669149716 2024-05-04T12:14:15.461 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid d9b985c4-0a0e-11ef-bc97-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-05-04T12:14:15.526 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:14:15 smithi146 ceph-mon[35311]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-04T12:14:15.871 INFO:teuthology.orchestra.run.smithi079.stdout:167503724552 2024-05-04T12:14:15.871 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid d9b985c4-0a0e-11ef-bc97-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-05-04T12:14:16.094 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:16.094+0000 7fa66e438700 1 -- 172.21.15.79:0/1164017223 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa668102760 msgr2=0x7fa66810ac50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:14:16.094 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:16.094+0000 7fa66e438700 1 --2- 172.21.15.79:0/1164017223 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa668102760 0x7fa66810ac50 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fa65c009940 tx=0x7fa65c009c50 comp rx=0 tx=0).stop 2024-05-04T12:14:16.094 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:16.094+0000 7fa66e438700 1 -- 172.21.15.79:0/1164017223 shutdown_connections 2024-05-04T12:14:16.094 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:16.094+0000 7fa66e438700 1 --2- 172.21.15.79:0/1164017223 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa668102760 0x7fa66810ac50 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:16.094 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:16.094+0000 7fa66e438700 1 --2- 172.21.15.79:0/1164017223 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa668101e40 0x7fa668102220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:16.094 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:16.094+0000 7fa66e438700 1 -- 172.21.15.79:0/1164017223 >> 172.21.15.79:0/1164017223 conn(0x7fa6680fb610 msgr2=0x7fa6680fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:14:16.095 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:16.094+0000 7fa66e438700 1 -- 172.21.15.79:0/1164017223 shutdown_connections 2024-05-04T12:14:16.095 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:16.095+0000 7fa66e438700 1 -- 172.21.15.79:0/1164017223 wait complete. 2024-05-04T12:14:16.095 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:16.095+0000 7fa66e438700 1 Processor -- start 2024-05-04T12:14:16.095 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:16.095+0000 7fa66e438700 1 -- start start 2024-05-04T12:14:16.096 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:16.096+0000 7fa66e438700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa668101e40 0x7fa66819c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:14:16.096 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:16.096+0000 7fa66e438700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa668102760 0x7fa66819cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:14:16.097 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:16.096+0000 7fa66e438700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa66819d220 con 0x7fa668102760 2024-05-04T12:14:16.097 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:16.096+0000 7fa66e438700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa668196720 con 0x7fa668101e40 2024-05-04T12:14:16.097 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:16.096+0000 7fa6677fe700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa668102760 0x7fa66819cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:14:16.097 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:16.096+0000 7fa6677fe700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa668102760 0x7fa66819cb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:53474/0 (socket says 172.21.15.79:53474) 2024-05-04T12:14:16.097 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:16.096+0000 7fa6677fe700 1 -- 172.21.15.79:0/2677930000 learned_addr learned my addr 172.21.15.79:0/2677930000 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:14:16.097 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:16.096+0000 7fa667fff700 1 --2- 172.21.15.79:0/2677930000 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa668101e40 0x7fa66819c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:14:16.097 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:16.096+0000 7fa6677fe700 1 -- 172.21.15.79:0/2677930000 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa668101e40 msgr2=0x7fa66819c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:14:16.097 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:16.096+0000 7fa6677fe700 1 --2- 172.21.15.79:0/2677930000 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa668101e40 0x7fa66819c650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:16.097 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:16.096+0000 7fa6677fe700 1 -- 172.21.15.79:0/2677930000 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa65c0095f0 con 0x7fa668102760 2024-05-04T12:14:16.097 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:16.096+0000 7fa6677fe700 1 --2- 172.21.15.79:0/2677930000 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa668102760 0x7fa66819cb90 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fa65c0055f0 tx=0x7fa65c005620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:14:16.097 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:16.096+0000 7fa6657fa700 1 -- 172.21.15.79:0/2677930000 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa65c01d070 con 0x7fa668102760 2024-05-04T12:14:16.097 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:16.097+0000 7fa66e438700 1 -- 172.21.15.79:0/2677930000 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6681969a0 con 0x7fa668102760 2024-05-04T12:14:16.097 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:16.097+0000 7fa6657fa700 1 -- 172.21.15.79:0/2677930000 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fa65c00f460 con 0x7fa668102760 2024-05-04T12:14:16.098 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:16.097+0000 7fa6657fa700 1 -- 172.21.15.79:0/2677930000 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa65c021640 con 0x7fa668102760 2024-05-04T12:14:16.098 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:16.097+0000 7fa66e438700 1 -- 172.21.15.79:0/2677930000 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa668196e90 con 0x7fa668102760 2024-05-04T12:14:16.098 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:16.098+0000 7fa66e438700 1 -- 172.21.15.79:0/2677930000 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa66810e0e0 con 0x7fa668102760 2024-05-04T12:14:16.099 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:16.099+0000 7fa6657fa700 1 -- 172.21.15.79:0/2677930000 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7fa65c00faf0 con 0x7fa668102760 2024-05-04T12:14:16.099 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:16.099+0000 7fa6657fa700 1 --2- 172.21.15.79:0/2677930000 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7fa65006c530 0x7fa65006e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:14:16.099 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:16.099+0000 7fa6657fa700 1 -- 172.21.15.79:0/2677930000 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fa65c08d290 con 0x7fa668102760 2024-05-04T12:14:16.099 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:16.099+0000 7fa667fff700 1 --2- 172.21.15.79:0/2677930000 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7fa65006c530 0x7fa65006e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:14:16.099 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:16.099+0000 7fa6657fa700 1 --2- 172.21.15.79:0/2677930000 >> [v2:172.21.15.79:6818/3776007511,v1:172.21.15.79:6819/3776007511] conn(0x7fa650072210 0x7fa650074630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:14:16.099 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:16.099+0000 7fa6657fa700 1 -- 172.21.15.79:0/2677930000 --> [v2:172.21.15.79:6818/3776007511,v1:172.21.15.79:6819/3776007511] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa650074ce0 con 0x7fa650072210 2024-05-04T12:14:16.099 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:16.099+0000 7fa6657fa700 1 -- 172.21.15.79:0/2677930000 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa65c08d640 con 0x7fa668102760 2024-05-04T12:14:16.099 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:16.100+0000 7fa66c9d5700 1 --2- 172.21.15.79:0/2677930000 >> [v2:172.21.15.79:6818/3776007511,v1:172.21.15.79:6819/3776007511] conn(0x7fa650072210 0x7fa650074630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:14:16.099 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:16.100+0000 7fa667fff700 1 --2- 172.21.15.79:0/2677930000 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7fa65006c530 0x7fa65006e9f0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fa658005950 tx=0x7fa65800a300 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:14:16.099 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:16.100+0000 7fa66c9d5700 1 --2- 172.21.15.79:0/2677930000 >> [v2:172.21.15.79:6818/3776007511,v1:172.21.15.79:6819/3776007511] conn(0x7fa650072210 0x7fa650074630 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:14:16.100 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:16.101+0000 7fa6657fa700 1 -- 172.21.15.79:0/2677930000 <== osd.2 v2:172.21.15.79:6818/3776007511 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fa650074ce0 con 0x7fa650072210 2024-05-04T12:14:16.117 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:16.118+0000 7fa66e438700 1 -- 172.21.15.79:0/2677930000 --> [v2:172.21.15.79:6818/3776007511,v1:172.21.15.79:6819/3776007511] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fa6680679c0 con 0x7fa650072210 2024-05-04T12:14:16.117 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:16.118+0000 7fa6657fa700 1 -- 172.21.15.79:0/2677930000 <== osd.2 v2:172.21.15.79:6818/3776007511 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fa6680679c0 con 0x7fa650072210 2024-05-04T12:14:16.119 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:16.119+0000 7fa66e438700 1 -- 172.21.15.79:0/2677930000 >> [v2:172.21.15.79:6818/3776007511,v1:172.21.15.79:6819/3776007511] conn(0x7fa650072210 msgr2=0x7fa650074630 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:14:16.119 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:16.119+0000 7fa66e438700 1 --2- 172.21.15.79:0/2677930000 >> [v2:172.21.15.79:6818/3776007511,v1:172.21.15.79:6819/3776007511] conn(0x7fa650072210 0x7fa650074630 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:16.119 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:16.119+0000 7fa66e438700 1 -- 172.21.15.79:0/2677930000 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7fa65006c530 msgr2=0x7fa65006e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:14:16.119 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:16.119+0000 7fa66e438700 1 --2- 172.21.15.79:0/2677930000 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7fa65006c530 0x7fa65006e9f0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fa658005950 tx=0x7fa65800a300 comp rx=0 tx=0).stop 2024-05-04T12:14:16.119 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:16.119+0000 7fa66e438700 1 -- 172.21.15.79:0/2677930000 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa668102760 msgr2=0x7fa66819cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:14:16.119 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:16.119+0000 7fa66e438700 1 --2- 172.21.15.79:0/2677930000 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa668102760 0x7fa66819cb90 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fa65c0055f0 tx=0x7fa65c005620 comp rx=0 tx=0).stop 2024-05-04T12:14:16.119 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:16.119+0000 7fa66e438700 1 -- 172.21.15.79:0/2677930000 shutdown_connections 2024-05-04T12:14:16.119 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:16.119+0000 7fa66e438700 1 --2- 172.21.15.79:0/2677930000 >> [v2:172.21.15.79:6818/3776007511,v1:172.21.15.79:6819/3776007511] conn(0x7fa650072210 0x7fa650074630 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:16.119 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:16.119+0000 7fa66e438700 1 --2- 172.21.15.79:0/2677930000 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7fa65006c530 0x7fa65006e9f0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:16.119 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:16.119+0000 7fa66e438700 1 --2- 172.21.15.79:0/2677930000 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa668101e40 0x7fa66819c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:16.119 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:16.119+0000 7fa66e438700 1 --2- 172.21.15.79:0/2677930000 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa668102760 0x7fa66819cb90 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:16.119 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:16.119+0000 7fa66e438700 1 -- 172.21.15.79:0/2677930000 >> 172.21.15.79:0/2677930000 conn(0x7fa6680fb610 msgr2=0x7fa668105490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:14:16.119 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:16.119+0000 7fa66e438700 1 -- 172.21.15.79:0/2677930000 shutdown_connections 2024-05-04T12:14:16.120 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:16.119+0000 7fa66e438700 1 -- 172.21.15.79:0/2677930000 wait complete. 2024-05-04T12:14:16.639 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:16.637+0000 7fbc255f3700 1 -- 172.21.15.79:0/1585148933 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbc200ffc20 msgr2=0x7fbc20100080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:14:16.639 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:16.637+0000 7fbc255f3700 1 --2- 172.21.15.79:0/1585148933 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbc200ffc20 0x7fbc20100080 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fbc14009910 tx=0x7fbc14009c20 comp rx=0 tx=0).stop 2024-05-04T12:14:16.639 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:16.637+0000 7fbc255f3700 1 -- 172.21.15.79:0/1585148933 shutdown_connections 2024-05-04T12:14:16.639 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:16.637+0000 7fbc255f3700 1 --2- 172.21.15.79:0/1585148933 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbc200ffc20 0x7fbc20100080 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:16.639 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:16.637+0000 7fbc255f3700 1 --2- 172.21.15.79:0/1585148933 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbc20105bd0 0x7fbc20105fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:16.639 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:16.637+0000 7fbc255f3700 1 -- 172.21.15.79:0/1585148933 >> 172.21.15.79:0/1585148933 conn(0x7fbc20075670 msgr2=0x7fbc20075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:14:16.639 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:16.637+0000 7fbc255f3700 1 -- 172.21.15.79:0/1585148933 shutdown_connections 2024-05-04T12:14:16.639 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:16.637+0000 7fbc255f3700 1 -- 172.21.15.79:0/1585148933 wait complete. 2024-05-04T12:14:16.639 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:16.638+0000 7fbc255f3700 1 Processor -- start 2024-05-04T12:14:16.640 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:16.638+0000 7fbc255f3700 1 -- start start 2024-05-04T12:14:16.640 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:16.638+0000 7fbc255f3700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbc200ffc20 0x7fbc20197700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:14:16.640 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:16.638+0000 7fbc255f3700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbc20105bd0 0x7fbc20197c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:14:16.640 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:16.638+0000 7fbc255f3700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc20198320 con 0x7fbc20105bd0 2024-05-04T12:14:16.640 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:16.638+0000 7fbc255f3700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc20198460 con 0x7fbc200ffc20 2024-05-04T12:14:16.640 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:16.638+0000 7fbc1e7fc700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbc20105bd0 0x7fbc20197c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:14:16.640 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:16.638+0000 7fbc1e7fc700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbc20105bd0 0x7fbc20197c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:53500/0 (socket says 172.21.15.79:53500) 2024-05-04T12:14:16.640 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:16.638+0000 7fbc1e7fc700 1 -- 172.21.15.79:0/3630374627 learned_addr learned my addr 172.21.15.79:0/3630374627 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:14:16.641 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:16.639+0000 7fbc1effd700 1 --2- 172.21.15.79:0/3630374627 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbc200ffc20 0x7fbc20197700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:14:16.641 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:16.639+0000 7fbc1e7fc700 1 -- 172.21.15.79:0/3630374627 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbc200ffc20 msgr2=0x7fbc20197700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:14:16.641 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:16.639+0000 7fbc1e7fc700 1 --2- 172.21.15.79:0/3630374627 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbc200ffc20 0x7fbc20197700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:16.641 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:16.639+0000 7fbc1e7fc700 1 -- 172.21.15.79:0/3630374627 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc140095f0 con 0x7fbc20105bd0 2024-05-04T12:14:16.641 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:16.639+0000 7fbc1e7fc700 1 --2- 172.21.15.79:0/3630374627 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbc20105bd0 0x7fbc20197c40 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fbc14006010 tx=0x7fbc1400bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:14:16.641 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:16.639+0000 7fbc07fff700 1 -- 172.21.15.79:0/3630374627 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbc1401d070 con 0x7fbc20105bd0 2024-05-04T12:14:16.641 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:16.639+0000 7fbc07fff700 1 -- 172.21.15.79:0/3630374627 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fbc14004fa0 con 0x7fbc20105bd0 2024-05-04T12:14:16.641 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:16.639+0000 7fbc255f3700 1 -- 172.21.15.79:0/3630374627 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc201a0770 con 0x7fbc20105bd0 2024-05-04T12:14:16.641 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:16.639+0000 7fbc07fff700 1 -- 172.21.15.79:0/3630374627 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbc1400f690 con 0x7fbc20105bd0 2024-05-04T12:14:16.641 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:16.639+0000 7fbc255f3700 1 -- 172.21.15.79:0/3630374627 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc201a0c30 con 0x7fbc20105bd0 2024-05-04T12:14:16.643 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:16.640+0000 7fbc255f3700 1 -- 172.21.15.79:0/3630374627 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fbc2010cfa0 con 0x7fbc20105bd0 2024-05-04T12:14:16.643 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:16.641+0000 7fbc07fff700 1 -- 172.21.15.79:0/3630374627 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7fbc14022470 con 0x7fbc20105bd0 2024-05-04T12:14:16.643 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:16.641+0000 7fbc07fff700 1 --2- 172.21.15.79:0/3630374627 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7fbc0806c4e0 0x7fbc0806e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:14:16.643 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:16.641+0000 7fbc07fff700 1 -- 172.21.15.79:0/3630374627 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fbc1408cbd0 con 0x7fbc20105bd0 2024-05-04T12:14:16.643 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:16.641+0000 7fbc1effd700 1 --2- 172.21.15.79:0/3630374627 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7fbc0806c4e0 0x7fbc0806e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:14:16.643 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:16.641+0000 7fbc07fff700 1 --2- 172.21.15.79:0/3630374627 >> [v2:172.21.15.79:6802/1309201220,v1:172.21.15.79:6803/1309201220] conn(0x7fbc080721c0 0x7fbc080745e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:14:16.643 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:16.641+0000 7fbc07fff700 1 -- 172.21.15.79:0/3630374627 --> [v2:172.21.15.79:6802/1309201220,v1:172.21.15.79:6803/1309201220] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbc08074c90 con 0x7fbc080721c0 2024-05-04T12:14:16.643 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:16.641+0000 7fbc07fff700 1 -- 172.21.15.79:0/3630374627 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbc140922b0 con 0x7fbc20105bd0 2024-05-04T12:14:16.643 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:16.641+0000 7fbc1f7fe700 1 --2- 172.21.15.79:0/3630374627 >> [v2:172.21.15.79:6802/1309201220,v1:172.21.15.79:6803/1309201220] conn(0x7fbc080721c0 0x7fbc080745e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:14:16.643 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:16.641+0000 7fbc1effd700 1 --2- 172.21.15.79:0/3630374627 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7fbc0806c4e0 0x7fbc0806e9a0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fbc10005950 tx=0x7fbc1000a300 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:14:16.643 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:16.642+0000 7fbc1f7fe700 1 --2- 172.21.15.79:0/3630374627 >> [v2:172.21.15.79:6802/1309201220,v1:172.21.15.79:6803/1309201220] conn(0x7fbc080721c0 0x7fbc080745e0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:14:16.644 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:16.642+0000 7fbc07fff700 1 -- 172.21.15.79:0/3630374627 <== osd.0 v2:172.21.15.79:6802/1309201220 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fbc08074c90 con 0x7fbc080721c0 2024-05-04T12:14:16.674 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:16.674+0000 7fbc255f3700 1 -- 172.21.15.79:0/3630374627 --> [v2:172.21.15.79:6802/1309201220,v1:172.21.15.79:6803/1309201220] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fbc2004f2e0 con 0x7fbc080721c0 2024-05-04T12:14:16.674 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:16.675+0000 7fbc07fff700 1 -- 172.21.15.79:0/3630374627 <== osd.0 v2:172.21.15.79:6802/1309201220 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fbc2004f2e0 con 0x7fbc080721c0 2024-05-04T12:14:16.675 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:16.675+0000 7fbc255f3700 1 -- 172.21.15.79:0/3630374627 >> [v2:172.21.15.79:6802/1309201220,v1:172.21.15.79:6803/1309201220] conn(0x7fbc080721c0 msgr2=0x7fbc080745e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:14:16.675 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:16.675+0000 7fbc255f3700 1 --2- 172.21.15.79:0/3630374627 >> [v2:172.21.15.79:6802/1309201220,v1:172.21.15.79:6803/1309201220] conn(0x7fbc080721c0 0x7fbc080745e0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:16.675 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:16.675+0000 7fbc255f3700 1 -- 172.21.15.79:0/3630374627 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7fbc0806c4e0 msgr2=0x7fbc0806e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:14:16.675 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:16.675+0000 7fbc255f3700 1 --2- 172.21.15.79:0/3630374627 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7fbc0806c4e0 0x7fbc0806e9a0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fbc10005950 tx=0x7fbc1000a300 comp rx=0 tx=0).stop 2024-05-04T12:14:16.675 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:16.675+0000 7fbc255f3700 1 -- 172.21.15.79:0/3630374627 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbc20105bd0 msgr2=0x7fbc20197c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:14:16.675 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:16.675+0000 7fbc255f3700 1 --2- 172.21.15.79:0/3630374627 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbc20105bd0 0x7fbc20197c40 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fbc14006010 tx=0x7fbc1400bd70 comp rx=0 tx=0).stop 2024-05-04T12:14:16.675 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:16.675+0000 7fbc255f3700 1 -- 172.21.15.79:0/3630374627 shutdown_connections 2024-05-04T12:14:16.675 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:16.675+0000 7fbc255f3700 1 --2- 172.21.15.79:0/3630374627 >> [v2:172.21.15.79:6802/1309201220,v1:172.21.15.79:6803/1309201220] conn(0x7fbc080721c0 0x7fbc080745e0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:16.676 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:16.675+0000 7fbc255f3700 1 --2- 172.21.15.79:0/3630374627 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7fbc0806c4e0 0x7fbc0806e9a0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:16.676 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:16.675+0000 7fbc255f3700 1 --2- 172.21.15.79:0/3630374627 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbc200ffc20 0x7fbc20197700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:16.676 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:16.675+0000 7fbc255f3700 1 --2- 172.21.15.79:0/3630374627 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbc20105bd0 0x7fbc20197c40 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:16.676 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:16.675+0000 7fbc255f3700 1 -- 172.21.15.79:0/3630374627 >> 172.21.15.79:0/3630374627 conn(0x7fbc20075670 msgr2=0x7fbc200fecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:14:16.676 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:16.675+0000 7fbc255f3700 1 -- 172.21.15.79:0/3630374627 shutdown_connections 2024-05-04T12:14:16.676 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:16.675+0000 7fbc255f3700 1 -- 172.21.15.79:0/3630374627 wait complete. 2024-05-04T12:14:17.218 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:14:17 smithi079 ceph-mon[27222]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-04T12:14:17.319 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:17.320+0000 7faa50a95700 1 -- 172.21.15.79:0/1628349434 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7faa4c102700 msgr2=0x7faa4c10abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:14:17.320 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:17.320+0000 7faa50a95700 1 --2- 172.21.15.79:0/1628349434 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7faa4c102700 0x7faa4c10abf0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7faa40009910 tx=0x7faa40009c20 comp rx=0 tx=0).stop 2024-05-04T12:14:17.320 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:17.320+0000 7faa50a95700 1 -- 172.21.15.79:0/1628349434 shutdown_connections 2024-05-04T12:14:17.320 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:17.320+0000 7faa50a95700 1 --2- 172.21.15.79:0/1628349434 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7faa4c102700 0x7faa4c10abf0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:17.320 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:17.320+0000 7faa50a95700 1 --2- 172.21.15.79:0/1628349434 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faa4c101de0 0x7faa4c1021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:17.320 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:17.320+0000 7faa50a95700 1 -- 172.21.15.79:0/1628349434 >> 172.21.15.79:0/1628349434 conn(0x7faa4c0fb610 msgr2=0x7faa4c0fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:14:17.320 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:17.320+0000 7faa50a95700 1 -- 172.21.15.79:0/1628349434 shutdown_connections 2024-05-04T12:14:17.320 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:17.320+0000 7faa50a95700 1 -- 172.21.15.79:0/1628349434 wait complete. 2024-05-04T12:14:17.322 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:17.321+0000 7faa50a95700 1 Processor -- start 2024-05-04T12:14:17.322 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:17.321+0000 7faa50a95700 1 -- start start 2024-05-04T12:14:17.322 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:17.321+0000 7faa50a95700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faa4c101de0 0x7faa4c0ffa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:14:17.323 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:17.321+0000 7faa50a95700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7faa4c102700 0x7faa4c0fff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:14:17.323 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:17.321+0000 7faa50a95700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa4c1004c0 con 0x7faa4c102700 2024-05-04T12:14:17.323 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:17.321+0000 7faa50a95700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa4c100600 con 0x7faa4c101de0 2024-05-04T12:14:17.323 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:17.322+0000 7faa49d9b700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7faa4c102700 0x7faa4c0fff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:14:17.323 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:17.322+0000 7faa49d9b700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7faa4c102700 0x7faa4c0fff80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:53514/0 (socket says 172.21.15.79:53514) 2024-05-04T12:14:17.323 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:17.322+0000 7faa49d9b700 1 -- 172.21.15.79:0/884787864 learned_addr learned my addr 172.21.15.79:0/884787864 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:14:17.323 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:17.322+0000 7faa4a59c700 1 --2- 172.21.15.79:0/884787864 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faa4c101de0 0x7faa4c0ffa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:14:17.323 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:17.322+0000 7faa49d9b700 1 -- 172.21.15.79:0/884787864 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faa4c101de0 msgr2=0x7faa4c0ffa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:14:17.323 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:17.322+0000 7faa49d9b700 1 --2- 172.21.15.79:0/884787864 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faa4c101de0 0x7faa4c0ffa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:17.323 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:17.322+0000 7faa49d9b700 1 -- 172.21.15.79:0/884787864 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa400095f0 con 0x7faa4c102700 2024-05-04T12:14:17.323 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:17.322+0000 7faa49d9b700 1 --2- 172.21.15.79:0/884787864 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7faa4c102700 0x7faa4c0fff80 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7faa400092e0 tx=0x7faa4000bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:14:17.324 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:17.322+0000 7faa3b7fe700 1 -- 172.21.15.79:0/884787864 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faa4001d070 con 0x7faa4c102700 2024-05-04T12:14:17.324 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:17.322+0000 7faa50a95700 1 -- 172.21.15.79:0/884787864 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa4c100880 con 0x7faa4c102700 2024-05-04T12:14:17.324 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:17.322+0000 7faa50a95700 1 -- 172.21.15.79:0/884787864 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa4c1a2d80 con 0x7faa4c102700 2024-05-04T12:14:17.324 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:17.322+0000 7faa3b7fe700 1 -- 172.21.15.79:0/884787864 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7faa4000f460 con 0x7faa4c102700 2024-05-04T12:14:17.324 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:17.322+0000 7faa3b7fe700 1 -- 172.21.15.79:0/884787864 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faa40017630 con 0x7faa4c102700 2024-05-04T12:14:17.324 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:17.323+0000 7faa50a95700 1 -- 172.21.15.79:0/884787864 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7faa2c000ff0 con 0x7faa4c102700 2024-05-04T12:14:17.325 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:17.324+0000 7faa3b7fe700 1 -- 172.21.15.79:0/884787864 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7faa400178a0 con 0x7faa4c102700 2024-05-04T12:14:17.325 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:17.324+0000 7faa3b7fe700 1 --2- 172.21.15.79:0/884787864 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7faa3406c4e0 0x7faa3406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:14:17.325 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:17.324+0000 7faa3b7fe700 1 -- 172.21.15.79:0/884787864 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7faa4008c630 con 0x7faa4c102700 2024-05-04T12:14:17.326 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:17.324+0000 7faa4a59c700 1 --2- 172.21.15.79:0/884787864 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7faa3406c4e0 0x7faa3406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:14:17.326 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:17.324+0000 7faa3b7fe700 1 --2- 172.21.15.79:0/884787864 >> [v2:172.21.15.79:6810/3443379349,v1:172.21.15.79:6811/3443379349] conn(0x7faa340721c0 0x7faa340745e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:14:17.326 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:17.324+0000 7faa3b7fe700 1 -- 172.21.15.79:0/884787864 --> [v2:172.21.15.79:6810/3443379349,v1:172.21.15.79:6811/3443379349] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7faa34074c90 con 0x7faa340721c0 2024-05-04T12:14:17.326 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:17.324+0000 7faa3b7fe700 1 -- 172.21.15.79:0/884787864 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7faa4008ed00 con 0x7faa4c102700 2024-05-04T12:14:17.326 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:17.325+0000 7faa4ad9d700 1 --2- 172.21.15.79:0/884787864 >> [v2:172.21.15.79:6810/3443379349,v1:172.21.15.79:6811/3443379349] conn(0x7faa340721c0 0x7faa340745e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:14:17.326 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:17.325+0000 7faa4a59c700 1 --2- 172.21.15.79:0/884787864 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7faa3406c4e0 0x7faa3406e9a0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7faa3c005950 tx=0x7faa3c00b410 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:14:17.326 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:17.325+0000 7faa4ad9d700 1 --2- 172.21.15.79:0/884787864 >> [v2:172.21.15.79:6810/3443379349,v1:172.21.15.79:6811/3443379349] conn(0x7faa340721c0 0x7faa340745e0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:14:17.326 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:17.325+0000 7faa3b7fe700 1 -- 172.21.15.79:0/884787864 <== osd.1 v2:172.21.15.79:6810/3443379349 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7faa34074c90 con 0x7faa340721c0 2024-05-04T12:14:17.340 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:17.341+0000 7faa50a95700 1 -- 172.21.15.79:0/884787864 --> [v2:172.21.15.79:6810/3443379349,v1:172.21.15.79:6811/3443379349] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7faa2c002ce0 con 0x7faa340721c0 2024-05-04T12:14:17.340 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:17.341+0000 7faa3b7fe700 1 -- 172.21.15.79:0/884787864 <== osd.1 v2:172.21.15.79:6810/3443379349 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7faa2c002ce0 con 0x7faa340721c0 2024-05-04T12:14:17.341 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:17.341+0000 7faa50a95700 1 -- 172.21.15.79:0/884787864 >> [v2:172.21.15.79:6810/3443379349,v1:172.21.15.79:6811/3443379349] conn(0x7faa340721c0 msgr2=0x7faa340745e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:14:17.341 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:17.341+0000 7faa50a95700 1 --2- 172.21.15.79:0/884787864 >> [v2:172.21.15.79:6810/3443379349,v1:172.21.15.79:6811/3443379349] conn(0x7faa340721c0 0x7faa340745e0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:17.341 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:17.341+0000 7faa50a95700 1 -- 172.21.15.79:0/884787864 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7faa3406c4e0 msgr2=0x7faa3406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:14:17.341 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:17.341+0000 7faa50a95700 1 --2- 172.21.15.79:0/884787864 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7faa3406c4e0 0x7faa3406e9a0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7faa3c005950 tx=0x7faa3c00b410 comp rx=0 tx=0).stop 2024-05-04T12:14:17.341 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:17.341+0000 7faa50a95700 1 -- 172.21.15.79:0/884787864 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7faa4c102700 msgr2=0x7faa4c0fff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:14:17.341 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:17.341+0000 7faa50a95700 1 --2- 172.21.15.79:0/884787864 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7faa4c102700 0x7faa4c0fff80 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7faa400092e0 tx=0x7faa4000bd70 comp rx=0 tx=0).stop 2024-05-04T12:14:17.341 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:17.341+0000 7faa50a95700 1 -- 172.21.15.79:0/884787864 shutdown_connections 2024-05-04T12:14:17.342 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:17.341+0000 7faa50a95700 1 --2- 172.21.15.79:0/884787864 >> [v2:172.21.15.79:6810/3443379349,v1:172.21.15.79:6811/3443379349] conn(0x7faa340721c0 0x7faa340745e0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:17.342 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:17.341+0000 7faa50a95700 1 --2- 172.21.15.79:0/884787864 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7faa3406c4e0 0x7faa3406e9a0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:17.342 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:17.341+0000 7faa50a95700 1 --2- 172.21.15.79:0/884787864 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faa4c101de0 0x7faa4c0ffa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:17.342 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:17.341+0000 7faa50a95700 1 --2- 172.21.15.79:0/884787864 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7faa4c102700 0x7faa4c0fff80 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:17.342 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:17.341+0000 7faa50a95700 1 -- 172.21.15.79:0/884787864 >> 172.21.15.79:0/884787864 conn(0x7faa4c0fb610 msgr2=0x7faa4c105430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:14:17.342 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:17.342+0000 7faa50a95700 1 -- 172.21.15.79:0/884787864 shutdown_connections 2024-05-04T12:14:17.342 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:17.342+0000 7faa50a95700 1 -- 172.21.15.79:0/884787864 wait complete. 2024-05-04T12:14:17.526 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:14:17 smithi146 ceph-mon[35311]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-04T12:14:17.996 INFO:teuthology.orchestra.run.smithi079.stdout:38654705701 2024-05-04T12:14:17.997 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid d9b985c4-0a0e-11ef-bc97-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-05-04T12:14:18.710 INFO:teuthology.orchestra.run.smithi079.stdout:81604378649 2024-05-04T12:14:18.710 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid d9b985c4-0a0e-11ef-bc97-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-05-04T12:14:19.151 INFO:teuthology.orchestra.run.smithi079.stdout:60129542174 2024-05-04T12:14:19.151 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid d9b985c4-0a0e-11ef-bc97-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-05-04T12:14:19.364 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:14:19 smithi079 ceph-mon[27222]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-04T12:14:19.392 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:19.393+0000 7fe8f8986700 1 -- 172.21.15.79:0/3647750803 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe8f4105bd0 msgr2=0x7fe8f4105fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:14:19.392 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:19.393+0000 7fe8f8986700 1 --2- 172.21.15.79:0/3647750803 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe8f4105bd0 0x7fe8f4105fb0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fe8e4009910 tx=0x7fe8e4009c20 comp rx=0 tx=0).stop 2024-05-04T12:14:19.392 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:19.393+0000 7fe8f8986700 1 -- 172.21.15.79:0/3647750803 shutdown_connections 2024-05-04T12:14:19.393 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:19.393+0000 7fe8f8986700 1 --2- 172.21.15.79:0/3647750803 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe8f40ffc20 0x7fe8f4100080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:19.393 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:19.393+0000 7fe8f8986700 1 --2- 172.21.15.79:0/3647750803 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe8f4105bd0 0x7fe8f4105fb0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:19.393 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:19.393+0000 7fe8f8986700 1 -- 172.21.15.79:0/3647750803 >> 172.21.15.79:0/3647750803 conn(0x7fe8f4075670 msgr2=0x7fe8f4075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:14:19.393 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:19.393+0000 7fe8f8986700 1 -- 172.21.15.79:0/3647750803 shutdown_connections 2024-05-04T12:14:19.393 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:19.393+0000 7fe8f8986700 1 -- 172.21.15.79:0/3647750803 wait complete. 2024-05-04T12:14:19.393 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:19.394+0000 7fe8f8986700 1 Processor -- start 2024-05-04T12:14:19.393 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:19.395+0000 7fe8f8986700 1 -- start start 2024-05-04T12:14:19.394 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:19.395+0000 7fe8f8986700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe8f40ffc20 0x7fe8f4194690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:14:19.394 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:19.395+0000 7fe8f8986700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe8f4105bd0 0x7fe8f4194bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:14:19.394 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:19.395+0000 7fe8f8986700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8f41952b0 con 0x7fe8f4105bd0 2024-05-04T12:14:19.395 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:19.395+0000 7fe8f8986700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8f4198e30 con 0x7fe8f40ffc20 2024-05-04T12:14:19.395 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:19.395+0000 7fe8f1d9b700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe8f4105bd0 0x7fe8f4194bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:14:19.395 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:19.395+0000 7fe8f259c700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe8f40ffc20 0x7fe8f4194690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:14:19.395 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:19.395+0000 7fe8f1d9b700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe8f4105bd0 0x7fe8f4194bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:53528/0 (socket says 172.21.15.79:53528) 2024-05-04T12:14:19.395 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:19.395+0000 7fe8f1d9b700 1 -- 172.21.15.79:0/3218581766 learned_addr learned my addr 172.21.15.79:0/3218581766 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:14:19.396 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:19.395+0000 7fe8f1d9b700 1 -- 172.21.15.79:0/3218581766 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe8f40ffc20 msgr2=0x7fe8f4194690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:14:19.396 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:19.395+0000 7fe8f1d9b700 1 --2- 172.21.15.79:0/3218581766 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe8f40ffc20 0x7fe8f4194690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:19.396 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:19.395+0000 7fe8f1d9b700 1 -- 172.21.15.79:0/3218581766 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8e40095f0 con 0x7fe8f4105bd0 2024-05-04T12:14:19.396 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:19.396+0000 7fe8f1d9b700 1 --2- 172.21.15.79:0/3218581766 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe8f4105bd0 0x7fe8f4194bd0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fe8e800d8d0 tx=0x7fe8e800dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:14:19.396 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:19.396+0000 7fe8e37fe700 1 -- 172.21.15.79:0/3218581766 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe8e80098a0 con 0x7fe8f4105bd0 2024-05-04T12:14:19.396 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:19.396+0000 7fe8e37fe700 1 -- 172.21.15.79:0/3218581766 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fe8e8010460 con 0x7fe8f4105bd0 2024-05-04T12:14:19.396 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:19.396+0000 7fe8f8986700 1 -- 172.21.15.79:0/3218581766 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8f4199110 con 0x7fe8f4105bd0 2024-05-04T12:14:19.397 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:19.396+0000 7fe8f8986700 1 -- 172.21.15.79:0/3218581766 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8f4199660 con 0x7fe8f4105bd0 2024-05-04T12:14:19.397 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:19.397+0000 7fe8e37fe700 1 -- 172.21.15.79:0/3218581766 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe8e800f5f0 con 0x7fe8f4105bd0 2024-05-04T12:14:19.397 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:19.397+0000 7fe8f8986700 1 -- 172.21.15.79:0/3218581766 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fe8d4000ff0 con 0x7fe8f4105bd0 2024-05-04T12:14:19.399 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:19.399+0000 7fe8e37fe700 1 -- 172.21.15.79:0/3218581766 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7fe8e8009a00 con 0x7fe8f4105bd0 2024-05-04T12:14:19.399 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:19.400+0000 7fe8e37fe700 1 --2- 172.21.15.79:0/3218581766 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7fe8dc06e610 0x7fe8dc070ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:14:19.399 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:19.400+0000 7fe8f259c700 1 --2- 172.21.15.79:0/3218581766 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7fe8dc06e610 0x7fe8dc070ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:14:19.399 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:19.400+0000 7fe8e37fe700 1 -- 172.21.15.79:0/3218581766 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fe8e808b430 con 0x7fe8f4105bd0 2024-05-04T12:14:19.400 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:19.400+0000 7fe8e37fe700 1 --2- 172.21.15.79:0/3218581766 >> [v2:172.21.15.146:6808/3913803877,v1:172.21.15.146:6809/3913803877] conn(0x7fe8dc074180 0x7fe8dc0765a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:14:19.400 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:19.400+0000 7fe8e37fe700 1 -- 172.21.15.79:0/3218581766 --> [v2:172.21.15.146:6808/3913803877,v1:172.21.15.146:6809/3913803877] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe8dc076c50 con 0x7fe8dc074180 2024-05-04T12:14:19.400 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:19.400+0000 7fe8e37fe700 1 -- 172.21.15.79:0/3218581766 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe8e808b7e0 con 0x7fe8f4105bd0 2024-05-04T12:14:19.400 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:19.400+0000 7fe8f2d9d700 1 --2- 172.21.15.79:0/3218581766 >> [v2:172.21.15.146:6808/3913803877,v1:172.21.15.146:6809/3913803877] conn(0x7fe8dc074180 0x7fe8dc0765a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:14:19.400 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:19.401+0000 7fe8f259c700 1 --2- 172.21.15.79:0/3218581766 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7fe8dc06e610 0x7fe8dc070ad0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fe8e40098e0 tx=0x7fe8e401a040 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:14:19.401 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:19.401+0000 7fe8f2d9d700 1 --2- 172.21.15.79:0/3218581766 >> [v2:172.21.15.146:6808/3913803877,v1:172.21.15.146:6809/3913803877] conn(0x7fe8dc074180 0x7fe8dc0765a0 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:14:19.401 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:19.402+0000 7fe8e37fe700 1 -- 172.21.15.79:0/3218581766 <== osd.4 v2:172.21.15.146:6808/3913803877 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fe8dc076c50 con 0x7fe8dc074180 2024-05-04T12:14:19.421 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:19.422+0000 7fe8f8986700 1 -- 172.21.15.79:0/3218581766 --> [v2:172.21.15.146:6808/3913803877,v1:172.21.15.146:6809/3913803877] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fe8d4002d70 con 0x7fe8dc074180 2024-05-04T12:14:19.421 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:19.423+0000 7fe8e37fe700 1 -- 172.21.15.79:0/3218581766 <== osd.4 v2:172.21.15.146:6808/3913803877 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fe8d4002d70 con 0x7fe8dc074180 2024-05-04T12:14:19.422 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:19.423+0000 7fe8f8986700 1 -- 172.21.15.79:0/3218581766 >> [v2:172.21.15.146:6808/3913803877,v1:172.21.15.146:6809/3913803877] conn(0x7fe8dc074180 msgr2=0x7fe8dc0765a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:14:19.422 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:19.423+0000 7fe8f8986700 1 --2- 172.21.15.79:0/3218581766 >> [v2:172.21.15.146:6808/3913803877,v1:172.21.15.146:6809/3913803877] conn(0x7fe8dc074180 0x7fe8dc0765a0 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:19.422 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:19.423+0000 7fe8f8986700 1 -- 172.21.15.79:0/3218581766 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7fe8dc06e610 msgr2=0x7fe8dc070ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:14:19.423 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:19.423+0000 7fe8f8986700 1 --2- 172.21.15.79:0/3218581766 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7fe8dc06e610 0x7fe8dc070ad0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fe8e40098e0 tx=0x7fe8e401a040 comp rx=0 tx=0).stop 2024-05-04T12:14:19.423 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:19.423+0000 7fe8f8986700 1 -- 172.21.15.79:0/3218581766 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe8f4105bd0 msgr2=0x7fe8f4194bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:14:19.423 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:19.423+0000 7fe8f8986700 1 --2- 172.21.15.79:0/3218581766 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe8f4105bd0 0x7fe8f4194bd0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fe8e800d8d0 tx=0x7fe8e800dbe0 comp rx=0 tx=0).stop 2024-05-04T12:14:19.423 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:19.423+0000 7fe8f8986700 1 -- 172.21.15.79:0/3218581766 shutdown_connections 2024-05-04T12:14:19.423 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:19.423+0000 7fe8f8986700 1 --2- 172.21.15.79:0/3218581766 >> [v2:172.21.15.146:6808/3913803877,v1:172.21.15.146:6809/3913803877] conn(0x7fe8dc074180 0x7fe8dc0765a0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:19.423 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:19.423+0000 7fe8f8986700 1 --2- 172.21.15.79:0/3218581766 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7fe8dc06e610 0x7fe8dc070ad0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:19.423 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:19.423+0000 7fe8f8986700 1 --2- 172.21.15.79:0/3218581766 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe8f40ffc20 0x7fe8f4194690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:19.423 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:19.423+0000 7fe8f8986700 1 --2- 172.21.15.79:0/3218581766 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe8f4105bd0 0x7fe8f4194bd0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:19.423 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:19.423+0000 7fe8f8986700 1 -- 172.21.15.79:0/3218581766 >> 172.21.15.79:0/3218581766 conn(0x7fe8f4075670 msgr2=0x7fe8f40febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:14:19.423 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:19.424+0000 7fe8f8986700 1 -- 172.21.15.79:0/3218581766 shutdown_connections 2024-05-04T12:14:19.423 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:19.424+0000 7fe8f8986700 1 -- 172.21.15.79:0/3218581766 wait complete. 2024-05-04T12:14:19.431 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/d9b985c4-0a0e-11ef-bc97-c7b262605968/mon.smithi079/config 2024-05-04T12:14:19.431 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/d9b985c4-0a0e-11ef-bc97-c7b262605968/mon.smithi079/config 2024-05-04T12:14:19.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:14:19 smithi146 ceph-mon[35311]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-04T12:14:20.149 INFO:teuthology.orchestra.run.smithi079.stdout:146028888079 2024-05-04T12:14:20.150 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid d9b985c4-0a0e-11ef-bc97-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-05-04T12:14:20.182 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/d9b985c4-0a0e-11ef-bc97-c7b262605968/mon.smithi079/config 2024-05-04T12:14:20.201 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/d9b985c4-0a0e-11ef-bc97-c7b262605968/mon.smithi079/config 2024-05-04T12:14:20.208 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/d9b985c4-0a0e-11ef-bc97-c7b262605968/mon.smithi079/config 2024-05-04T12:14:21.195 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:14:21 smithi079 ceph-mon[27222]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-04T12:14:21.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:14:21 smithi146 ceph-mon[35311]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-04T12:14:23.164 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:14:23 smithi079 ceph-mon[27222]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-04T12:14:23.526 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:14:23 smithi146 ceph-mon[35311]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-04T12:14:24.403 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:24.387+0000 7f0e902e1700 1 -- 172.21.15.79:0/2520309558 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0e88105350 msgr2=0x7f0e88105730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:14:24.403 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:24.387+0000 7f0e902e1700 1 --2- 172.21.15.79:0/2520309558 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0e88105350 0x7f0e88105730 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f0e8000c210 tx=0x7f0e8000c520 comp rx=0 tx=0).stop 2024-05-04T12:14:24.403 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:24.387+0000 7f0e902e1700 1 -- 172.21.15.79:0/2520309558 shutdown_connections 2024-05-04T12:14:24.403 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:24.387+0000 7f0e902e1700 1 --2- 172.21.15.79:0/2520309558 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0e880ff340 0x7f0e880ff7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:24.403 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:24.387+0000 7f0e902e1700 1 --2- 172.21.15.79:0/2520309558 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0e88105350 0x7f0e88105730 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:24.403 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:24.387+0000 7f0e902e1700 1 -- 172.21.15.79:0/2520309558 >> 172.21.15.79:0/2520309558 conn(0x7f0e88074e30 msgr2=0x7f0e88075240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:14:24.404 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:24.387+0000 7f0e902e1700 1 -- 172.21.15.79:0/2520309558 shutdown_connections 2024-05-04T12:14:24.404 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:24.387+0000 7f0e902e1700 1 -- 172.21.15.79:0/2520309558 wait complete. 2024-05-04T12:14:24.404 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:24.388+0000 7f0e902e1700 1 Processor -- start 2024-05-04T12:14:24.404 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:24.388+0000 7f0e902e1700 1 -- start start 2024-05-04T12:14:24.404 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:24.388+0000 7f0e902e1700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0e880ff340 0x7f0e8819c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:14:24.404 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:24.388+0000 7f0e902e1700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0e88105350 0x7f0e8819cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:14:24.404 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:24.388+0000 7f0e902e1700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e8819d220 con 0x7f0e88105350 2024-05-04T12:14:24.404 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:24.388+0000 7f0e902e1700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e881966d0 con 0x7f0e880ff340 2024-05-04T12:14:24.404 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:24.388+0000 7f0e8d87c700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0e88105350 0x7f0e8819cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:14:24.404 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:24.388+0000 7f0e8d87c700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0e88105350 0x7f0e8819cb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:49746/0 (socket says 172.21.15.79:49746) 2024-05-04T12:14:24.404 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:24.388+0000 7f0e8d87c700 1 -- 172.21.15.79:0/2726728600 learned_addr learned my addr 172.21.15.79:0/2726728600 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:14:24.404 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:24.388+0000 7f0e8e07d700 1 --2- 172.21.15.79:0/2726728600 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0e880ff340 0x7f0e8819c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:14:24.404 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:24.389+0000 7f0e8d87c700 1 -- 172.21.15.79:0/2726728600 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0e880ff340 msgr2=0x7f0e8819c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:14:24.404 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:24.389+0000 7f0e8d87c700 1 --2- 172.21.15.79:0/2726728600 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0e880ff340 0x7f0e8819c650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:24.404 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:24.389+0000 7f0e8d87c700 1 -- 172.21.15.79:0/2726728600 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e8000ae10 con 0x7f0e88105350 2024-05-04T12:14:24.405 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:24.389+0000 7f0e8d87c700 1 --2- 172.21.15.79:0/2726728600 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0e88105350 0x7f0e8819cb90 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f0e7c00d8d0 tx=0x7f0e7c00dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:14:24.405 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:24.389+0000 7f0e877fe700 1 -- 172.21.15.79:0/2726728600 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0e7c0098a0 con 0x7f0e88105350 2024-05-04T12:14:24.405 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:24.389+0000 7f0e877fe700 1 -- 172.21.15.79:0/2726728600 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f0e7c010460 con 0x7f0e88105350 2024-05-04T12:14:24.405 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:24.389+0000 7f0e902e1700 1 -- 172.21.15.79:0/2726728600 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e881969b0 con 0x7f0e88105350 2024-05-04T12:14:24.405 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:24.389+0000 7f0e877fe700 1 -- 172.21.15.79:0/2726728600 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0e7c00f5f0 con 0x7f0e88105350 2024-05-04T12:14:24.405 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:24.389+0000 7f0e902e1700 1 -- 172.21.15.79:0/2726728600 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e88196f00 con 0x7f0e88105350 2024-05-04T12:14:24.405 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:24.390+0000 7f0e902e1700 1 -- 172.21.15.79:0/2726728600 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e88109880 con 0x7f0e88105350 2024-05-04T12:14:24.405 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:24.390+0000 7f0e877fe700 1 -- 172.21.15.79:0/2726728600 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f0e7c009a00 con 0x7f0e88105350 2024-05-04T12:14:24.405 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:24.391+0000 7f0e877fe700 1 --2- 172.21.15.79:0/2726728600 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f0e7006c4e0 0x7f0e7006e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:14:24.405 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:24.391+0000 7f0e8e07d700 1 --2- 172.21.15.79:0/2726728600 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f0e7006c4e0 0x7f0e7006e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:14:24.405 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:24.391+0000 7f0e877fe700 1 -- 172.21.15.79:0/2726728600 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f0e7c08b430 con 0x7f0e88105350 2024-05-04T12:14:24.405 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:24.391+0000 7f0e8e07d700 1 --2- 172.21.15.79:0/2726728600 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f0e7006c4e0 0x7f0e7006e9a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f0e80000f80 tx=0x7f0e800060c0 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:14:24.405 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:24.395+0000 7f0e877fe700 1 -- 172.21.15.79:0/2726728600 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f0e7c059980 con 0x7f0e88105350 2024-05-04T12:14:24.591 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:24.592+0000 7f0e902e1700 1 -- 172.21.15.79:0/2726728600 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f0e8804f2e0 con 0x7f0e88105350 2024-05-04T12:14:24.591 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:24.592+0000 7f0e877fe700 1 -- 172.21.15.79:0/2726728600 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f0e7c059510 con 0x7f0e88105350 2024-05-04T12:14:24.592 INFO:teuthology.orchestra.run.smithi079.stdout:81604378651 2024-05-04T12:14:24.595 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:24.595+0000 7f0e902e1700 1 -- 172.21.15.79:0/2726728600 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f0e7006c4e0 msgr2=0x7f0e7006e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:14:24.595 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:24.595+0000 7f0e902e1700 1 --2- 172.21.15.79:0/2726728600 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f0e7006c4e0 0x7f0e7006e9a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f0e80000f80 tx=0x7f0e800060c0 comp rx=0 tx=0).stop 2024-05-04T12:14:24.595 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:24.595+0000 7f0e902e1700 1 -- 172.21.15.79:0/2726728600 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0e88105350 msgr2=0x7f0e8819cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:14:24.595 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:24.595+0000 7f0e902e1700 1 --2- 172.21.15.79:0/2726728600 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0e88105350 0x7f0e8819cb90 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f0e7c00d8d0 tx=0x7f0e7c00dbe0 comp rx=0 tx=0).stop 2024-05-04T12:14:24.595 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:24.595+0000 7f0e902e1700 1 -- 172.21.15.79:0/2726728600 shutdown_connections 2024-05-04T12:14:24.595 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:24.595+0000 7f0e902e1700 1 --2- 172.21.15.79:0/2726728600 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f0e7006c4e0 0x7f0e7006e9a0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:24.595 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:24.595+0000 7f0e902e1700 1 --2- 172.21.15.79:0/2726728600 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0e880ff340 0x7f0e8819c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:24.596 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:24.595+0000 7f0e902e1700 1 --2- 172.21.15.79:0/2726728600 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0e88105350 0x7f0e8819cb90 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:24.596 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:24.595+0000 7f0e902e1700 1 -- 172.21.15.79:0/2726728600 >> 172.21.15.79:0/2726728600 conn(0x7f0e88074e30 msgr2=0x7f0e881038b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:14:24.596 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:24.596+0000 7f0e902e1700 1 -- 172.21.15.79:0/2726728600 shutdown_connections 2024-05-04T12:14:24.596 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:24.596+0000 7f0e902e1700 1 -- 172.21.15.79:0/2726728600 wait complete. 2024-05-04T12:14:24.629 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:24.609+0000 7f818058c700 1 -- 172.21.15.79:0/1334906438 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8178101840 msgr2=0x7f8178105890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:14:24.629 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:24.609+0000 7f818058c700 1 --2- 172.21.15.79:0/1334906438 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8178101840 0x7f8178105890 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f8168009910 tx=0x7f8168009c20 comp rx=0 tx=0).stop 2024-05-04T12:14:24.629 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:24.609+0000 7f818058c700 1 -- 172.21.15.79:0/1334906438 shutdown_connections 2024-05-04T12:14:24.629 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:24.609+0000 7f818058c700 1 --2- 172.21.15.79:0/1334906438 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8178101840 0x7f8178105890 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:24.629 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:24.609+0000 7f818058c700 1 --2- 172.21.15.79:0/1334906438 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8178100e90 0x7f8178101270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:24.629 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:24.609+0000 7f818058c700 1 -- 172.21.15.79:0/1334906438 >> 172.21.15.79:0/1334906438 conn(0x7f81780fc740 msgr2=0x7f81780feb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:14:24.629 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:24.609+0000 7f818058c700 1 -- 172.21.15.79:0/1334906438 shutdown_connections 2024-05-04T12:14:24.629 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:24.609+0000 7f818058c700 1 -- 172.21.15.79:0/1334906438 wait complete. 2024-05-04T12:14:24.630 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:24.609+0000 7f818058c700 1 Processor -- start 2024-05-04T12:14:24.630 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:24.609+0000 7f818058c700 1 -- start start 2024-05-04T12:14:24.630 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:24.610+0000 7f818058c700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8178100e90 0x7f817819c670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:14:24.630 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:24.610+0000 7f818058c700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8178101840 0x7f817819cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:14:24.630 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:24.610+0000 7f818058c700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f817819d240 con 0x7f8178100e90 2024-05-04T12:14:24.630 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:24.610+0000 7f818058c700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81781966f0 con 0x7f8178101840 2024-05-04T12:14:24.630 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:24.610+0000 7f817e328700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8178100e90 0x7f817819c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:14:24.630 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:24.610+0000 7f817e328700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8178100e90 0x7f817819c670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:49766/0 (socket says 172.21.15.79:49766) 2024-05-04T12:14:24.630 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:24.610+0000 7f817e328700 1 -- 172.21.15.79:0/3390183997 learned_addr learned my addr 172.21.15.79:0/3390183997 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:14:24.630 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:24.610+0000 7f817db27700 1 --2- 172.21.15.79:0/3390183997 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8178101840 0x7f817819cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:14:24.630 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:24.610+0000 7f817e328700 1 -- 172.21.15.79:0/3390183997 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8178101840 msgr2=0x7f817819cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:14:24.630 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:24.610+0000 7f817e328700 1 --2- 172.21.15.79:0/3390183997 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8178101840 0x7f817819cbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:24.631 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:24.610+0000 7f817e328700 1 -- 172.21.15.79:0/3390183997 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81680095f0 con 0x7f8178100e90 2024-05-04T12:14:24.631 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:24.610+0000 7f817e328700 1 --2- 172.21.15.79:0/3390183997 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8178100e90 0x7f817819c670 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f817400d900 tx=0x7f817400dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:14:24.631 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:24.610+0000 7f816f7fe700 1 -- 172.21.15.79:0/3390183997 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f81740041f0 con 0x7f8178100e90 2024-05-04T12:14:24.631 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:24.610+0000 7f818058c700 1 -- 172.21.15.79:0/3390183997 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81781969d0 con 0x7f8178100e90 2024-05-04T12:14:24.631 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:24.610+0000 7f816f7fe700 1 -- 172.21.15.79:0/3390183997 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f8174004ce0 con 0x7f8178100e90 2024-05-04T12:14:24.631 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:24.610+0000 7f816f7fe700 1 -- 172.21.15.79:0/3390183997 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f817400b750 con 0x7f8178100e90 2024-05-04T12:14:24.631 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:24.611+0000 7f818058c700 1 -- 172.21.15.79:0/3390183997 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8178196f20 con 0x7f8178100e90 2024-05-04T12:14:24.631 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:24.612+0000 7f818058c700 1 -- 172.21.15.79:0/3390183997 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8178109290 con 0x7f8178100e90 2024-05-04T12:14:24.631 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:24.612+0000 7f816f7fe700 1 -- 172.21.15.79:0/3390183997 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f8174004e50 con 0x7f8178100e90 2024-05-04T12:14:24.631 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:24.612+0000 7f816f7fe700 1 --2- 172.21.15.79:0/3390183997 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f816406c4e0 0x7f816406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:14:24.631 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:24.613+0000 7f816f7fe700 1 -- 172.21.15.79:0/3390183997 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f817408b490 con 0x7f8178100e90 2024-05-04T12:14:24.631 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:24.613+0000 7f817db27700 1 --2- 172.21.15.79:0/3390183997 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f816406c4e0 0x7f816406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:14:24.631 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:24.613+0000 7f817db27700 1 --2- 172.21.15.79:0/3390183997 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f816406c4e0 0x7f816406e9a0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f8178197b60 tx=0x7f816801a040 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:14:24.631 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:24.617+0000 7f816f7fe700 1 -- 172.21.15.79:0/3390183997 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f81740599a0 con 0x7f8178100e90 2024-05-04T12:14:24.810 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:24.810+0000 7f818058c700 1 -- 172.21.15.79:0/3390183997 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f8178197d00 con 0x7f8178100e90 2024-05-04T12:14:24.810 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:24.811+0000 7f816f7fe700 1 -- 172.21.15.79:0/3390183997 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f8178197d00 con 0x7f8178100e90 2024-05-04T12:14:24.810 INFO:teuthology.orchestra.run.smithi079.stdout:38654705702 2024-05-04T12:14:24.812 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:24.813+0000 7f818058c700 1 -- 172.21.15.79:0/3390183997 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f816406c4e0 msgr2=0x7f816406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:14:24.812 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:24.813+0000 7f818058c700 1 --2- 172.21.15.79:0/3390183997 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f816406c4e0 0x7f816406e9a0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f8178197b60 tx=0x7f816801a040 comp rx=0 tx=0).stop 2024-05-04T12:14:24.812 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:24.813+0000 7f818058c700 1 -- 172.21.15.79:0/3390183997 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8178100e90 msgr2=0x7f817819c670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:14:24.813 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:24.813+0000 7f818058c700 1 --2- 172.21.15.79:0/3390183997 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8178100e90 0x7f817819c670 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f817400d900 tx=0x7f817400dc10 comp rx=0 tx=0).stop 2024-05-04T12:14:24.813 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:24.813+0000 7f818058c700 1 -- 172.21.15.79:0/3390183997 shutdown_connections 2024-05-04T12:14:24.813 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:24.813+0000 7f818058c700 1 --2- 172.21.15.79:0/3390183997 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f816406c4e0 0x7f816406e9a0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:24.813 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:24.813+0000 7f818058c700 1 --2- 172.21.15.79:0/3390183997 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8178100e90 0x7f817819c670 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:24.813 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:24.813+0000 7f818058c700 1 --2- 172.21.15.79:0/3390183997 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8178101840 0x7f817819cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:24.813 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:24.813+0000 7f818058c700 1 -- 172.21.15.79:0/3390183997 >> 172.21.15.79:0/3390183997 conn(0x7f81780fc740 msgr2=0x7f81780fddd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:14:24.813 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:24.813+0000 7f818058c700 1 -- 172.21.15.79:0/3390183997 shutdown_connections 2024-05-04T12:14:24.813 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:24.813+0000 7f818058c700 1 -- 172.21.15.79:0/3390183997 wait complete. 2024-05-04T12:14:25.195 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:14:25 smithi079 ceph-mon[27222]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-04T12:14:25.195 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:14:25 smithi079 ceph-mon[27222]: from='client.? 172.21.15.79:0/2726728600' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-05-04T12:14:25.195 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:14:25 smithi079 ceph-mon[27222]: from='client.? 172.21.15.79:0/3390183997' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-05-04T12:14:25.279 INFO:tasks.cephadm.ceph_manager.ceph:need seq 81604378649 got 81604378651 for osd.2 2024-05-04T12:14:25.280 DEBUG:teuthology.parallel:result is None 2024-05-04T12:14:25.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:14:25 smithi146 ceph-mon[35311]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-04T12:14:25.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:14:25 smithi146 ceph-mon[35311]: from='client.? 172.21.15.79:0/2726728600' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-05-04T12:14:25.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:14:25 smithi146 ceph-mon[35311]: from='client.? 172.21.15.79:0/3390183997' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-05-04T12:14:26.271 INFO:tasks.cephadm.ceph_manager.ceph:need seq 38654705701 got 38654705702 for osd.0 2024-05-04T12:14:26.271 DEBUG:teuthology.parallel:result is None 2024-05-04T12:14:26.492 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/d9b985c4-0a0e-11ef-bc97-c7b262605968/mon.smithi079/config 2024-05-04T12:14:27.412 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:14:27 smithi146 ceph-mon[35311]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-04T12:14:27.412 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:14:27 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-04T12:14:27.437 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:14:27 smithi079 ceph-mon[27222]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-04T12:14:27.437 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:14:27 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-04T12:14:29.279 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:29.278+0000 7f66ba7b4700 1 -- 172.21.15.79:0/1075138606 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f66b40ffda0 msgr2=0x7f66b4100180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:14:29.279 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:29.278+0000 7f66ba7b4700 1 --2- 172.21.15.79:0/1075138606 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f66b40ffda0 0x7f66b4100180 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f66a4009910 tx=0x7f66a4009c20 comp rx=0 tx=0).stop 2024-05-04T12:14:29.279 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:29.278+0000 7f66ba7b4700 1 -- 172.21.15.79:0/1075138606 shutdown_connections 2024-05-04T12:14:29.279 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:29.278+0000 7f66ba7b4700 1 --2- 172.21.15.79:0/1075138606 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f66b4100750 0x7f66b41047a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:29.279 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:29.278+0000 7f66ba7b4700 1 --2- 172.21.15.79:0/1075138606 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f66b40ffda0 0x7f66b4100180 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:29.279 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:29.278+0000 7f66ba7b4700 1 -- 172.21.15.79:0/1075138606 >> 172.21.15.79:0/1075138606 conn(0x7f66b40fb650 msgr2=0x7f66b40fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:14:29.280 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:29.278+0000 7f66ba7b4700 1 -- 172.21.15.79:0/1075138606 shutdown_connections 2024-05-04T12:14:29.280 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:29.278+0000 7f66ba7b4700 1 -- 172.21.15.79:0/1075138606 wait complete. 2024-05-04T12:14:29.280 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:29.279+0000 7f66ba7b4700 1 Processor -- start 2024-05-04T12:14:29.280 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:29.279+0000 7f66ba7b4700 1 -- start start 2024-05-04T12:14:29.280 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:29.279+0000 7f66ba7b4700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f66b40ffda0 0x7f66b4196980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:14:29.280 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:29.279+0000 7f66ba7b4700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f66b4100750 0x7f66b4196ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:14:29.280 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:29.279+0000 7f66ba7b4700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66b4197550 con 0x7f66b40ffda0 2024-05-04T12:14:29.280 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:29.279+0000 7f66ba7b4700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66b419b360 con 0x7f66b4100750 2024-05-04T12:14:29.280 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:29.279+0000 7f66b3fff700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f66b40ffda0 0x7f66b4196980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:14:29.280 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:29.279+0000 7f66b3fff700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f66b40ffda0 0x7f66b4196980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:49780/0 (socket says 172.21.15.79:49780) 2024-05-04T12:14:29.280 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:29.279+0000 7f66b3fff700 1 -- 172.21.15.79:0/3224901176 learned_addr learned my addr 172.21.15.79:0/3224901176 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:14:29.280 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:29.279+0000 7f66b37fe700 1 --2- 172.21.15.79:0/3224901176 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f66b4100750 0x7f66b4196ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:14:29.280 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:29.279+0000 7f66b3fff700 1 -- 172.21.15.79:0/3224901176 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f66b4100750 msgr2=0x7f66b4196ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:14:29.281 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:29.279+0000 7f66b3fff700 1 --2- 172.21.15.79:0/3224901176 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f66b4100750 0x7f66b4196ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:29.281 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:29.279+0000 7f66b3fff700 1 -- 172.21.15.79:0/3224901176 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66a40095f0 con 0x7f66b40ffda0 2024-05-04T12:14:29.281 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:29.279+0000 7f66b37fe700 1 --2- 172.21.15.79:0/3224901176 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f66b4100750 0x7f66b4196ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-04T12:14:29.281 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:29.280+0000 7f66b3fff700 1 --2- 172.21.15.79:0/3224901176 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f66b40ffda0 0x7f66b4196980 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f66a4006010 tx=0x7f66a400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:14:29.281 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:29.280+0000 7f66b17fa700 1 -- 172.21.15.79:0/3224901176 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f66a401d070 con 0x7f66b40ffda0 2024-05-04T12:14:29.281 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:29.280+0000 7f66b17fa700 1 -- 172.21.15.79:0/3224901176 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f66a400f460 con 0x7f66b40ffda0 2024-05-04T12:14:29.281 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:29.280+0000 7f66ba7b4700 1 -- 172.21.15.79:0/3224901176 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66b419b5e0 con 0x7f66b40ffda0 2024-05-04T12:14:29.281 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:29.280+0000 7f66b17fa700 1 -- 172.21.15.79:0/3224901176 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f66a4021640 con 0x7f66b40ffda0 2024-05-04T12:14:29.281 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:29.280+0000 7f66ba7b4700 1 -- 172.21.15.79:0/3224901176 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66b419bad0 con 0x7f66b40ffda0 2024-05-04T12:14:29.282 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:29.281+0000 7f66ba7b4700 1 -- 172.21.15.79:0/3224901176 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66b404f2e0 con 0x7f66b40ffda0 2024-05-04T12:14:29.282 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:29.281+0000 7f66b17fa700 1 -- 172.21.15.79:0/3224901176 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f66a400faf0 con 0x7f66b40ffda0 2024-05-04T12:14:29.282 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:29.282+0000 7f66b17fa700 1 --2- 172.21.15.79:0/3224901176 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f669c06c4e0 0x7f669c06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:14:29.282 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:29.282+0000 7f66b17fa700 1 -- 172.21.15.79:0/3224901176 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f66a408c870 con 0x7f66b40ffda0 2024-05-04T12:14:29.282 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:29.282+0000 7f66b37fe700 1 --2- 172.21.15.79:0/3224901176 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f669c06c4e0 0x7f669c06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:14:29.282 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:29.282+0000 7f66b37fe700 1 --2- 172.21.15.79:0/3224901176 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f669c06c4e0 0x7f669c06e9a0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f66b4197fa0 tx=0x7f66a8008040 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:14:29.285 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:29.286+0000 7f66b17fa700 1 -- 172.21.15.79:0/3224901176 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f66a405ae20 con 0x7f66b40ffda0 2024-05-04T12:14:29.369 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:14:29 smithi079 ceph-mon[27222]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-04T12:14:29.470 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:29.470+0000 7fb4ba135700 1 -- 172.21.15.79:0/2702540150 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb4b4101850 msgr2=0x7fb4b41058a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:14:29.470 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:29.471+0000 7fb4ba135700 1 --2- 172.21.15.79:0/2702540150 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb4b4101850 0x7fb4b41058a0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fb4a4009910 tx=0x7fb4a4009c20 comp rx=0 tx=0).stop 2024-05-04T12:14:29.470 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:29.471+0000 7fb4ba135700 1 -- 172.21.15.79:0/2702540150 shutdown_connections 2024-05-04T12:14:29.470 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:29.471+0000 7fb4ba135700 1 --2- 172.21.15.79:0/2702540150 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb4b4101850 0x7fb4b41058a0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:29.470 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:29.471+0000 7fb4ba135700 1 --2- 172.21.15.79:0/2702540150 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb4b4100ea0 0x7fb4b4101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:29.470 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:29.471+0000 7fb4ba135700 1 -- 172.21.15.79:0/2702540150 >> 172.21.15.79:0/2702540150 conn(0x7fb4b40fc750 msgr2=0x7fb4b40feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:14:29.470 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:29.471+0000 7fb4ba135700 1 -- 172.21.15.79:0/2702540150 shutdown_connections 2024-05-04T12:14:29.470 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:29.471+0000 7fb4ba135700 1 -- 172.21.15.79:0/2702540150 wait complete. 2024-05-04T12:14:29.472 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:29.471+0000 7fb4ba135700 1 Processor -- start 2024-05-04T12:14:29.472 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:29.472+0000 7fb4ba135700 1 -- start start 2024-05-04T12:14:29.472 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:29.472+0000 7fb4ba135700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb4b4100ea0 0x7fb4b419c640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:14:29.472 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:29.472+0000 7fb4ba135700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb4b4101850 0x7fb4b419cb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:14:29.472 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:29.472+0000 7fb4ba135700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4b419d210 con 0x7fb4b4100ea0 2024-05-04T12:14:29.472 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:29.472+0000 7fb4ba135700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4b41966c0 con 0x7fb4b4101850 2024-05-04T12:14:29.472 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:29.472+0000 7fb4b37fe700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb4b4100ea0 0x7fb4b419c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:14:29.473 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:29.472+0000 7fb4b37fe700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb4b4100ea0 0x7fb4b419c640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:49798/0 (socket says 172.21.15.79:49798) 2024-05-04T12:14:29.473 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:29.472+0000 7fb4b37fe700 1 -- 172.21.15.79:0/2107967286 learned_addr learned my addr 172.21.15.79:0/2107967286 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:14:29.473 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:29.472+0000 7fb4b2ffd700 1 --2- 172.21.15.79:0/2107967286 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb4b4101850 0x7fb4b419cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:14:29.473 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:29.472+0000 7fb4b37fe700 1 -- 172.21.15.79:0/2107967286 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb4b4101850 msgr2=0x7fb4b419cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:14:29.473 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:29.472+0000 7fb4b37fe700 1 --2- 172.21.15.79:0/2107967286 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb4b4101850 0x7fb4b419cb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:29.473 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:29.472+0000 7fb4b37fe700 1 -- 172.21.15.79:0/2107967286 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4a40095f0 con 0x7fb4b4100ea0 2024-05-04T12:14:29.473 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:29.473+0000 7fb4b37fe700 1 --2- 172.21.15.79:0/2107967286 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb4b4100ea0 0x7fb4b419c640 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fb4a800d8d0 tx=0x7fb4a800dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:14:29.473 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:29.473+0000 7fb4b0ff9700 1 -- 172.21.15.79:0/2107967286 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb4a80098a0 con 0x7fb4b4100ea0 2024-05-04T12:14:29.473 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:29.473+0000 7fb4ba135700 1 -- 172.21.15.79:0/2107967286 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4b41969a0 con 0x7fb4b4100ea0 2024-05-04T12:14:29.473 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:29.473+0000 7fb4b0ff9700 1 -- 172.21.15.79:0/2107967286 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fb4a8010460 con 0x7fb4b4100ea0 2024-05-04T12:14:29.473 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:29.473+0000 7fb4b0ff9700 1 -- 172.21.15.79:0/2107967286 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb4a800f5f0 con 0x7fb4b4100ea0 2024-05-04T12:14:29.473 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:29.473+0000 7fb4ba135700 1 -- 172.21.15.79:0/2107967286 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4b4196ef0 con 0x7fb4b4100ea0 2024-05-04T12:14:29.474 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:29.474+0000 7fb4ba135700 1 -- 172.21.15.79:0/2107967286 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4b41092a0 con 0x7fb4b4100ea0 2024-05-04T12:14:29.474 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:29.474+0000 7fb4b0ff9700 1 -- 172.21.15.79:0/2107967286 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7fb4a8009a00 con 0x7fb4b4100ea0 2024-05-04T12:14:29.474 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:29.475+0000 7fb4b0ff9700 1 --2- 172.21.15.79:0/2107967286 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7fb49c06c4e0 0x7fb49c06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:14:29.474 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:29.475+0000 7fb4b0ff9700 1 -- 172.21.15.79:0/2107967286 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fb4a808b2d0 con 0x7fb4b4100ea0 2024-05-04T12:14:29.474 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:29.475+0000 7fb4b2ffd700 1 --2- 172.21.15.79:0/2107967286 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7fb49c06c4e0 0x7fb49c06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:14:29.475 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:29.475+0000 7fb4b2ffd700 1 --2- 172.21.15.79:0/2107967286 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7fb49c06c4e0 0x7fb49c06e9a0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fb4b4197b30 tx=0x7fb4a401a040 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:14:29.478 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:29.479+0000 7fb4b0ff9700 1 -- 172.21.15.79:0/2107967286 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fb4a8059900 con 0x7fb4b4100ea0 2024-05-04T12:14:29.481 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:29.482+0000 7f66ba7b4700 1 -- 172.21.15.79:0/3224901176 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f66b419bcf0 con 0x7f66b40ffda0 2024-05-04T12:14:29.481 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:29.482+0000 7f66b17fa700 1 -- 172.21.15.79:0/3224901176 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f66a405a9b0 con 0x7f66b40ffda0 2024-05-04T12:14:29.481 INFO:teuthology.orchestra.run.smithi079.stdout:60129542177 2024-05-04T12:14:29.485 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:29.485+0000 7f66ba7b4700 1 -- 172.21.15.79:0/3224901176 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f669c06c4e0 msgr2=0x7f669c06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:14:29.485 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:29.485+0000 7f66ba7b4700 1 --2- 172.21.15.79:0/3224901176 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f669c06c4e0 0x7f669c06e9a0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f66b4197fa0 tx=0x7f66a8008040 comp rx=0 tx=0).stop 2024-05-04T12:14:29.485 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:29.485+0000 7f66ba7b4700 1 -- 172.21.15.79:0/3224901176 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f66b40ffda0 msgr2=0x7f66b4196980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:14:29.485 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:29.485+0000 7f66ba7b4700 1 --2- 172.21.15.79:0/3224901176 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f66b40ffda0 0x7f66b4196980 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f66a4006010 tx=0x7f66a400be20 comp rx=0 tx=0).stop 2024-05-04T12:14:29.485 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:29.485+0000 7f66ba7b4700 1 -- 172.21.15.79:0/3224901176 shutdown_connections 2024-05-04T12:14:29.485 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:29.485+0000 7f66ba7b4700 1 --2- 172.21.15.79:0/3224901176 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f669c06c4e0 0x7f669c06e9a0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:29.485 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:29.485+0000 7f66ba7b4700 1 --2- 172.21.15.79:0/3224901176 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f66b40ffda0 0x7f66b4196980 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:29.485 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:29.485+0000 7f66ba7b4700 1 --2- 172.21.15.79:0/3224901176 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f66b4100750 0x7f66b4196ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:29.485 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:29.485+0000 7f66ba7b4700 1 -- 172.21.15.79:0/3224901176 >> 172.21.15.79:0/3224901176 conn(0x7f66b40fb650 msgr2=0x7f66b4104000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:14:29.485 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:29.485+0000 7f66ba7b4700 1 -- 172.21.15.79:0/3224901176 shutdown_connections 2024-05-04T12:14:29.485 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:29.485+0000 7f66ba7b4700 1 -- 172.21.15.79:0/3224901176 wait complete. 2024-05-04T12:14:29.526 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:14:29 smithi146 ceph-mon[35311]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-04T12:14:29.689 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:29.690+0000 7fb4ba135700 1 -- 172.21.15.79:0/2107967286 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fb4b4197cd0 con 0x7fb4b4100ea0 2024-05-04T12:14:29.689 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:29.690+0000 7fb4b0ff9700 1 -- 172.21.15.79:0/2107967286 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fb4b4197cd0 con 0x7fb4b4100ea0 2024-05-04T12:14:29.689 INFO:teuthology.orchestra.run.smithi079.stdout:111669149719 2024-05-04T12:14:29.692 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:29.692+0000 7fb4ba135700 1 -- 172.21.15.79:0/2107967286 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7fb49c06c4e0 msgr2=0x7fb49c06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:14:29.692 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:29.692+0000 7fb4ba135700 1 --2- 172.21.15.79:0/2107967286 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7fb49c06c4e0 0x7fb49c06e9a0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fb4b4197b30 tx=0x7fb4a401a040 comp rx=0 tx=0).stop 2024-05-04T12:14:29.692 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:29.692+0000 7fb4ba135700 1 -- 172.21.15.79:0/2107967286 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb4b4100ea0 msgr2=0x7fb4b419c640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:14:29.692 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:29.692+0000 7fb4ba135700 1 --2- 172.21.15.79:0/2107967286 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb4b4100ea0 0x7fb4b419c640 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fb4a800d8d0 tx=0x7fb4a800dbe0 comp rx=0 tx=0).stop 2024-05-04T12:14:29.692 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:29.692+0000 7fb4ba135700 1 -- 172.21.15.79:0/2107967286 shutdown_connections 2024-05-04T12:14:29.692 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:29.692+0000 7fb4ba135700 1 --2- 172.21.15.79:0/2107967286 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7fb49c06c4e0 0x7fb49c06e9a0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:29.692 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:29.693+0000 7fb4ba135700 1 --2- 172.21.15.79:0/2107967286 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb4b4100ea0 0x7fb4b419c640 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:29.692 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:29.693+0000 7fb4ba135700 1 --2- 172.21.15.79:0/2107967286 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb4b4101850 0x7fb4b419cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:29.692 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:29.693+0000 7fb4ba135700 1 -- 172.21.15.79:0/2107967286 >> 172.21.15.79:0/2107967286 conn(0x7fb4b40fc750 msgr2=0x7fb4b40feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:14:29.692 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:29.693+0000 7fb4ba135700 1 -- 172.21.15.79:0/2107967286 shutdown_connections 2024-05-04T12:14:29.693 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:29.693+0000 7fb4ba135700 1 -- 172.21.15.79:0/2107967286 wait complete. 2024-05-04T12:14:30.163 INFO:tasks.cephadm.ceph_manager.ceph:need seq 60129542174 got 60129542177 for osd.1 2024-05-04T12:14:30.164 DEBUG:teuthology.parallel:result is None 2024-05-04T12:14:30.415 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:14:30 smithi079 ceph-mon[27222]: from='client.? 172.21.15.79:0/3224901176' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-05-04T12:14:30.415 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:14:30 smithi079 ceph-mon[27222]: from='client.? 172.21.15.79:0/2107967286' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-05-04T12:14:30.526 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:14:30 smithi146 ceph-mon[35311]: from='client.? 172.21.15.79:0/3224901176' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-05-04T12:14:30.526 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:14:30 smithi146 ceph-mon[35311]: from='client.? 172.21.15.79:0/2107967286' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-05-04T12:14:31.138 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149716 got 111669149719 for osd.3 2024-05-04T12:14:31.139 DEBUG:teuthology.parallel:result is None 2024-05-04T12:14:31.364 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:14:31 smithi079 ceph-mon[27222]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-04T12:14:31.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:14:31 smithi146 ceph-mon[35311]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-04T12:14:31.600 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:31.597+0000 7fa02a014700 1 -- 172.21.15.79:0/3793108467 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa024105bd0 msgr2=0x7fa024105fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:14:31.600 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:31.597+0000 7fa02a014700 1 --2- 172.21.15.79:0/3793108467 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa024105bd0 0x7fa024105fb0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fa018009910 tx=0x7fa018009c20 comp rx=0 tx=0).stop 2024-05-04T12:14:31.600 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:31.597+0000 7fa02a014700 1 -- 172.21.15.79:0/3793108467 shutdown_connections 2024-05-04T12:14:31.600 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:31.597+0000 7fa02a014700 1 --2- 172.21.15.79:0/3793108467 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa0240ffc20 0x7fa024100080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:31.600 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:31.597+0000 7fa02a014700 1 --2- 172.21.15.79:0/3793108467 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa024105bd0 0x7fa024105fb0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:31.600 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:31.597+0000 7fa02a014700 1 -- 172.21.15.79:0/3793108467 >> 172.21.15.79:0/3793108467 conn(0x7fa024075670 msgr2=0x7fa024075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:14:31.600 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:31.597+0000 7fa02a014700 1 -- 172.21.15.79:0/3793108467 shutdown_connections 2024-05-04T12:14:31.600 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:31.597+0000 7fa02a014700 1 -- 172.21.15.79:0/3793108467 wait complete. 2024-05-04T12:14:31.601 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:31.598+0000 7fa02a014700 1 Processor -- start 2024-05-04T12:14:31.601 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:31.598+0000 7fa02a014700 1 -- start start 2024-05-04T12:14:31.601 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:31.598+0000 7fa02a014700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa0240ffc20 0x7fa024195550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:14:31.601 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:31.598+0000 7fa02a014700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa024105bd0 0x7fa024195a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:14:31.601 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:31.598+0000 7fa02a014700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa024195fd0 con 0x7fa0240ffc20 2024-05-04T12:14:31.601 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:31.598+0000 7fa02a014700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa024196140 con 0x7fa024105bd0 2024-05-04T12:14:31.601 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:31.598+0000 7fa0237fe700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa0240ffc20 0x7fa024195550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:14:31.601 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:31.598+0000 7fa0237fe700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa0240ffc20 0x7fa024195550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:49812/0 (socket says 172.21.15.79:49812) 2024-05-04T12:14:31.601 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:31.598+0000 7fa0237fe700 1 -- 172.21.15.79:0/2707933273 learned_addr learned my addr 172.21.15.79:0/2707933273 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:14:31.601 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:31.598+0000 7fa022ffd700 1 --2- 172.21.15.79:0/2707933273 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa024105bd0 0x7fa024195a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:14:31.601 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:31.598+0000 7fa0237fe700 1 -- 172.21.15.79:0/2707933273 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa024105bd0 msgr2=0x7fa024195a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:14:31.601 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:31.598+0000 7fa0237fe700 1 --2- 172.21.15.79:0/2707933273 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa024105bd0 0x7fa024195a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:31.601 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:31.598+0000 7fa0237fe700 1 -- 172.21.15.79:0/2707933273 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0180095f0 con 0x7fa0240ffc20 2024-05-04T12:14:31.602 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:31.598+0000 7fa022ffd700 1 --2- 172.21.15.79:0/2707933273 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa024105bd0 0x7fa024195a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-04T12:14:31.602 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:31.599+0000 7fa0237fe700 1 --2- 172.21.15.79:0/2707933273 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa0240ffc20 0x7fa024195550 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fa018006010 tx=0x7fa01800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:14:31.602 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:31.599+0000 7fa020ff9700 1 -- 172.21.15.79:0/2707933273 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa01801d070 con 0x7fa0240ffc20 2024-05-04T12:14:31.602 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:31.599+0000 7fa020ff9700 1 -- 172.21.15.79:0/2707933273 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fa01800f460 con 0x7fa0240ffc20 2024-05-04T12:14:31.602 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:31.599+0000 7fa02a014700 1 -- 172.21.15.79:0/2707933273 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0241963c0 con 0x7fa0240ffc20 2024-05-04T12:14:31.602 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:31.599+0000 7fa020ff9700 1 -- 172.21.15.79:0/2707933273 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa018021640 con 0x7fa0240ffc20 2024-05-04T12:14:31.602 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:31.599+0000 7fa02a014700 1 -- 172.21.15.79:0/2707933273 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0241a2df0 con 0x7fa0240ffc20 2024-05-04T12:14:31.602 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:31.600+0000 7fa02a014700 1 -- 172.21.15.79:0/2707933273 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0241080f0 con 0x7fa0240ffc20 2024-05-04T12:14:31.602 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:31.600+0000 7fa020ff9700 1 -- 172.21.15.79:0/2707933273 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7fa01800faf0 con 0x7fa0240ffc20 2024-05-04T12:14:31.602 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:31.601+0000 7fa020ff9700 1 --2- 172.21.15.79:0/2707933273 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7fa00c06c4e0 0x7fa00c06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:14:31.602 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:31.601+0000 7fa020ff9700 1 -- 172.21.15.79:0/2707933273 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fa01808c6e0 con 0x7fa0240ffc20 2024-05-04T12:14:31.603 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:31.601+0000 7fa022ffd700 1 --2- 172.21.15.79:0/2707933273 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7fa00c06c4e0 0x7fa00c06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:14:31.603 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:31.601+0000 7fa022ffd700 1 --2- 172.21.15.79:0/2707933273 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7fa00c06c4e0 0x7fa00c06e9a0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fa0241976b0 tx=0x7fa014008040 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:14:31.604 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:31.605+0000 7fa020ff9700 1 -- 172.21.15.79:0/2707933273 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fa018057200 con 0x7fa0240ffc20 2024-05-04T12:14:31.801 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:31.801+0000 7fa02a014700 1 -- 172.21.15.79:0/2707933273 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7fa02404f2e0 con 0x7fa0240ffc20 2024-05-04T12:14:31.801 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:31.802+0000 7fa020ff9700 1 -- 172.21.15.79:0/2707933273 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fa01805a820 con 0x7fa0240ffc20 2024-05-04T12:14:31.801 INFO:teuthology.orchestra.run.smithi079.stdout:146028888081 2024-05-04T12:14:31.803 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:31.804+0000 7fa02a014700 1 -- 172.21.15.79:0/2707933273 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7fa00c06c4e0 msgr2=0x7fa00c06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:14:31.803 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:31.804+0000 7fa02a014700 1 --2- 172.21.15.79:0/2707933273 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7fa00c06c4e0 0x7fa00c06e9a0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fa0241976b0 tx=0x7fa014008040 comp rx=0 tx=0).stop 2024-05-04T12:14:31.803 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:31.804+0000 7fa02a014700 1 -- 172.21.15.79:0/2707933273 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa0240ffc20 msgr2=0x7fa024195550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:14:31.804 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:31.804+0000 7fa02a014700 1 --2- 172.21.15.79:0/2707933273 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa0240ffc20 0x7fa024195550 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fa018006010 tx=0x7fa01800be20 comp rx=0 tx=0).stop 2024-05-04T12:14:31.804 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:31.804+0000 7fa02a014700 1 -- 172.21.15.79:0/2707933273 shutdown_connections 2024-05-04T12:14:31.804 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:31.804+0000 7fa02a014700 1 --2- 172.21.15.79:0/2707933273 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7fa00c06c4e0 0x7fa00c06e9a0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:31.804 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:31.804+0000 7fa02a014700 1 --2- 172.21.15.79:0/2707933273 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa0240ffc20 0x7fa024195550 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:31.804 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:31.804+0000 7fa02a014700 1 --2- 172.21.15.79:0/2707933273 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa024105bd0 0x7fa024195a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:31.804 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:31.804+0000 7fa02a014700 1 -- 172.21.15.79:0/2707933273 >> 172.21.15.79:0/2707933273 conn(0x7fa024075670 msgr2=0x7fa0240fec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:14:31.804 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:31.804+0000 7fa02a014700 1 -- 172.21.15.79:0/2707933273 shutdown_connections 2024-05-04T12:14:31.804 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:31.804+0000 7fa02a014700 1 -- 172.21.15.79:0/2707933273 wait complete. 2024-05-04T12:14:31.835 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:31.836+0000 7f586cee9700 1 -- 172.21.15.79:0/3771983125 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f58680ff5a0 msgr2=0x7f58680ff980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:14:31.836 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:31.836+0000 7f586cee9700 1 --2- 172.21.15.79:0/3771983125 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f58680ff5a0 0x7f58680ff980 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f5858009910 tx=0x7f5858009c20 comp rx=0 tx=0).stop 2024-05-04T12:14:31.836 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:31.836+0000 7f586cee9700 1 -- 172.21.15.79:0/3771983125 shutdown_connections 2024-05-04T12:14:31.836 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:31.836+0000 7f586cee9700 1 --2- 172.21.15.79:0/3771983125 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f58680fff50 0x7f5868105840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:31.836 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:31.836+0000 7f586cee9700 1 --2- 172.21.15.79:0/3771983125 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f58680ff5a0 0x7f58680ff980 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:31.836 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:31.836+0000 7f586cee9700 1 -- 172.21.15.79:0/3771983125 >> 172.21.15.79:0/3771983125 conn(0x7f5868074e30 msgr2=0x7f5868075240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:14:31.836 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:31.836+0000 7f586cee9700 1 -- 172.21.15.79:0/3771983125 shutdown_connections 2024-05-04T12:14:31.836 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:31.837+0000 7f586cee9700 1 -- 172.21.15.79:0/3771983125 wait complete. 2024-05-04T12:14:31.838 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:31.837+0000 7f586cee9700 1 Processor -- start 2024-05-04T12:14:31.838 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:31.838+0000 7f586cee9700 1 -- start start 2024-05-04T12:14:31.838 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:31.838+0000 7f586cee9700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f58680ff5a0 0x7f586819c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:14:31.838 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:31.838+0000 7f586cee9700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f58680fff50 0x7f586819cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:14:31.838 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:31.838+0000 7f586cee9700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f586819d250 con 0x7f58680ff5a0 2024-05-04T12:14:31.838 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:31.838+0000 7f586cee9700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5868196700 con 0x7f58680fff50 2024-05-04T12:14:31.838 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:31.838+0000 7f586659c700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f58680ff5a0 0x7f586819c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:14:31.838 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:31.838+0000 7f586659c700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f58680ff5a0 0x7f586819c680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:49828/0 (socket says 172.21.15.79:49828) 2024-05-04T12:14:31.838 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:31.838+0000 7f586659c700 1 -- 172.21.15.79:0/2604977833 learned_addr learned my addr 172.21.15.79:0/2604977833 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:14:31.838 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:31.838+0000 7f585ffff700 1 --2- 172.21.15.79:0/2604977833 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f58680fff50 0x7f586819cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:14:31.839 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:31.838+0000 7f586659c700 1 -- 172.21.15.79:0/2604977833 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f58680fff50 msgr2=0x7f586819cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:14:31.839 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:31.838+0000 7f586659c700 1 --2- 172.21.15.79:0/2604977833 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f58680fff50 0x7f586819cbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:31.839 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:31.838+0000 7f586659c700 1 -- 172.21.15.79:0/2604977833 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58580095f0 con 0x7f58680ff5a0 2024-05-04T12:14:31.839 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:31.838+0000 7f586659c700 1 --2- 172.21.15.79:0/2604977833 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f58680ff5a0 0x7f586819c680 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f5858006010 tx=0x7f585800bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:14:31.839 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:31.839+0000 7f585f7fe700 1 -- 172.21.15.79:0/2604977833 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f585801d070 con 0x7f58680ff5a0 2024-05-04T12:14:31.839 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:31.839+0000 7f585f7fe700 1 -- 172.21.15.79:0/2604977833 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f585800f460 con 0x7f58680ff5a0 2024-05-04T12:14:31.839 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:31.839+0000 7f585f7fe700 1 -- 172.21.15.79:0/2604977833 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5858017630 con 0x7f58680ff5a0 2024-05-04T12:14:31.839 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:31.839+0000 7f586cee9700 1 -- 172.21.15.79:0/2604977833 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5868196980 con 0x7f58680ff5a0 2024-05-04T12:14:31.839 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:31.839+0000 7f586cee9700 1 -- 172.21.15.79:0/2604977833 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5868196e70 con 0x7f58680ff5a0 2024-05-04T12:14:31.840 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:31.840+0000 7f586cee9700 1 -- 172.21.15.79:0/2604977833 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5868109270 con 0x7f58680ff5a0 2024-05-04T12:14:31.840 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:31.840+0000 7f585f7fe700 1 -- 172.21.15.79:0/2604977833 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f5858003a60 con 0x7f58680ff5a0 2024-05-04T12:14:31.840 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:31.841+0000 7f585f7fe700 1 --2- 172.21.15.79:0/2604977833 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f585406c4e0 0x7f585406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:14:31.841 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:31.841+0000 7f585f7fe700 1 -- 172.21.15.79:0/2604977833 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f585808c850 con 0x7f58680ff5a0 2024-05-04T12:14:31.841 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:31.841+0000 7f585ffff700 1 --2- 172.21.15.79:0/2604977833 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f585406c4e0 0x7f585406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:14:31.841 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:31.841+0000 7f585ffff700 1 --2- 172.21.15.79:0/2604977833 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f585406c4e0 0x7f585406e9a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f5868197b70 tx=0x7f585000b410 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:14:31.844 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:31.845+0000 7f585f7fe700 1 -- 172.21.15.79:0/2604977833 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f585805ae80 con 0x7f58680ff5a0 2024-05-04T12:14:32.060 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:32.061+0000 7f586cee9700 1 -- 172.21.15.79:0/2604977833 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f5868197d10 con 0x7f58680ff5a0 2024-05-04T12:14:32.060 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:32.061+0000 7f585f7fe700 1 -- 172.21.15.79:0/2604977833 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f5868197d10 con 0x7f58680ff5a0 2024-05-04T12:14:32.060 INFO:teuthology.orchestra.run.smithi079.stdout:167503724555 2024-05-04T12:14:32.063 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:32.064+0000 7f586cee9700 1 -- 172.21.15.79:0/2604977833 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f585406c4e0 msgr2=0x7f585406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:14:32.063 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:32.064+0000 7f586cee9700 1 --2- 172.21.15.79:0/2604977833 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f585406c4e0 0x7f585406e9a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f5868197b70 tx=0x7f585000b410 comp rx=0 tx=0).stop 2024-05-04T12:14:32.064 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:32.064+0000 7f586cee9700 1 -- 172.21.15.79:0/2604977833 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f58680ff5a0 msgr2=0x7f586819c680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:14:32.064 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:32.064+0000 7f586cee9700 1 --2- 172.21.15.79:0/2604977833 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f58680ff5a0 0x7f586819c680 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f5858006010 tx=0x7f585800bd70 comp rx=0 tx=0).stop 2024-05-04T12:14:32.064 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:32.064+0000 7f586cee9700 1 -- 172.21.15.79:0/2604977833 shutdown_connections 2024-05-04T12:14:32.064 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:32.064+0000 7f586cee9700 1 --2- 172.21.15.79:0/2604977833 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f585406c4e0 0x7f585406e9a0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:32.064 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:32.064+0000 7f586cee9700 1 --2- 172.21.15.79:0/2604977833 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f58680ff5a0 0x7f586819c680 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:32.064 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:32.064+0000 7f586cee9700 1 --2- 172.21.15.79:0/2604977833 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f58680fff50 0x7f586819cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:32.064 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:32.064+0000 7f586cee9700 1 -- 172.21.15.79:0/2604977833 >> 172.21.15.79:0/2604977833 conn(0x7f5868074e30 msgr2=0x7f58680fdea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:14:32.064 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:32.064+0000 7f586cee9700 1 -- 172.21.15.79:0/2604977833 shutdown_connections 2024-05-04T12:14:32.064 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:32.064+0000 7f586cee9700 1 -- 172.21.15.79:0/2604977833 wait complete. 2024-05-04T12:14:32.358 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:14:32 smithi079 ceph-mon[27222]: from='client.? 172.21.15.79:0/2707933273' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-05-04T12:14:32.359 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:14:32 smithi079 ceph-mon[27222]: from='client.? 172.21.15.79:0/2604977833' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-05-04T12:14:32.450 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:14:32 smithi146 ceph-mon[35311]: from='client.? 172.21.15.79:0/2707933273' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-05-04T12:14:32.451 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:14:32 smithi146 ceph-mon[35311]: from='client.? 172.21.15.79:0/2604977833' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-05-04T12:14:32.626 INFO:tasks.cephadm.ceph_manager.ceph:need seq 146028888079 got 146028888081 for osd.4 2024-05-04T12:14:32.626 DEBUG:teuthology.parallel:result is None 2024-05-04T12:14:33.020 INFO:tasks.cephadm.ceph_manager.ceph:need seq 167503724552 got 167503724555 for osd.5 2024-05-04T12:14:33.020 DEBUG:teuthology.parallel:result is None 2024-05-04T12:14:33.020 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-05-04T12:14:33.021 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid d9b985c4-0a0e-11ef-bc97-c7b262605968 -- ceph pg dump --format=json 2024-05-04T12:14:33.233 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/d9b985c4-0a0e-11ef-bc97-c7b262605968/mon.smithi079/config 2024-05-04T12:14:33.333 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:14:33 smithi079 ceph-mon[27222]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-04T12:14:33.526 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:14:33 smithi146 ceph-mon[35311]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-04T12:14:34.725 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:34.725+0000 7f93d6ce6700 1 -- 172.21.15.79:0/3762093335 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f93d00fff50 msgr2=0x7f93d0105840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:14:34.725 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:34.725+0000 7f93d6ce6700 1 --2- 172.21.15.79:0/3762093335 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f93d00fff50 0x7f93d0105840 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f93bc009910 tx=0x7f93bc009c20 comp rx=0 tx=0).stop 2024-05-04T12:14:34.725 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:34.725+0000 7f93d6ce6700 1 -- 172.21.15.79:0/3762093335 shutdown_connections 2024-05-04T12:14:34.725 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:34.725+0000 7f93d6ce6700 1 --2- 172.21.15.79:0/3762093335 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f93d00fff50 0x7f93d0105840 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:34.725 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:34.725+0000 7f93d6ce6700 1 --2- 172.21.15.79:0/3762093335 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f93d00ff5a0 0x7f93d00ff980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:34.725 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:34.725+0000 7f93d6ce6700 1 -- 172.21.15.79:0/3762093335 >> 172.21.15.79:0/3762093335 conn(0x7f93d0074e30 msgr2=0x7f93d0075240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:14:34.725 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:34.725+0000 7f93d6ce6700 1 -- 172.21.15.79:0/3762093335 shutdown_connections 2024-05-04T12:14:34.725 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:34.725+0000 7f93d6ce6700 1 -- 172.21.15.79:0/3762093335 wait complete. 2024-05-04T12:14:34.727 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:34.726+0000 7f93d6ce6700 1 Processor -- start 2024-05-04T12:14:34.727 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:34.726+0000 7f93d6ce6700 1 -- start start 2024-05-04T12:14:34.727 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:34.726+0000 7f93d6ce6700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f93d00ff5a0 0x7f93d019c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:14:34.727 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:34.726+0000 7f93d6ce6700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f93d00fff50 0x7f93d019cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:14:34.727 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:34.726+0000 7f93d6ce6700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93d019d2c0 con 0x7f93d00ff5a0 2024-05-04T12:14:34.727 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:34.726+0000 7f93d6ce6700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93d0196770 con 0x7f93d00fff50 2024-05-04T12:14:34.727 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:34.726+0000 7f93d4a82700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f93d00ff5a0 0x7f93d019c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:14:34.727 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:34.726+0000 7f93d4a82700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f93d00ff5a0 0x7f93d019c6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:59208/0 (socket says 172.21.15.79:59208) 2024-05-04T12:14:34.728 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:34.726+0000 7f93d4a82700 1 -- 172.21.15.79:0/3217018100 learned_addr learned my addr 172.21.15.79:0/3217018100 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:14:34.728 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:34.727+0000 7f93d4a82700 1 -- 172.21.15.79:0/3217018100 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f93d00fff50 msgr2=0x7f93d019cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:14:34.728 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:34.727+0000 7f93d4a82700 1 --2- 172.21.15.79:0/3217018100 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f93d00fff50 0x7f93d019cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:34.728 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:34.727+0000 7f93d4a82700 1 -- 172.21.15.79:0/3217018100 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93bc0095f0 con 0x7f93d00ff5a0 2024-05-04T12:14:34.728 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:34.727+0000 7f93d4a82700 1 --2- 172.21.15.79:0/3217018100 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f93d00ff5a0 0x7f93d019c6f0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f93c400b4b0 tx=0x7f93c400b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:14:34.728 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:34.727+0000 7f93cdffb700 1 -- 172.21.15.79:0/3217018100 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f93c4011800 con 0x7f93d00ff5a0 2024-05-04T12:14:34.728 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:34.727+0000 7f93cdffb700 1 -- 172.21.15.79:0/3217018100 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f93c4011e10 con 0x7f93d00ff5a0 2024-05-04T12:14:34.728 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:34.727+0000 7f93d6ce6700 1 -- 172.21.15.79:0/3217018100 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93d0196a50 con 0x7f93d00ff5a0 2024-05-04T12:14:34.728 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:34.727+0000 7f93cdffb700 1 -- 172.21.15.79:0/3217018100 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f93c400f2e0 con 0x7f93d00ff5a0 2024-05-04T12:14:34.728 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:34.727+0000 7f93d6ce6700 1 -- 172.21.15.79:0/3217018100 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93d0196fa0 con 0x7f93d00ff5a0 2024-05-04T12:14:34.728 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:34.728+0000 7f93d6ce6700 1 -- 172.21.15.79:0/3217018100 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93d0109200 con 0x7f93d00ff5a0 2024-05-04T12:14:34.729 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:34.728+0000 7f93cdffb700 1 -- 172.21.15.79:0/3217018100 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f93c4010380 con 0x7f93d00ff5a0 2024-05-04T12:14:34.729 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:34.729+0000 7f93cdffb700 1 --2- 172.21.15.79:0/3217018100 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f93c006c4e0 0x7f93c006e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:14:34.729 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:34.729+0000 7f93cdffb700 1 -- 172.21.15.79:0/3217018100 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f93c408b170 con 0x7f93d00ff5a0 2024-05-04T12:14:34.729 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:34.729+0000 7f93cffff700 1 --2- 172.21.15.79:0/3217018100 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f93c006c4e0 0x7f93c006e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:14:34.729 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:34.729+0000 7f93cffff700 1 --2- 172.21.15.79:0/3217018100 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f93c006c4e0 0x7f93c006e9a0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f93bc00b5c0 tx=0x7f93bc005960 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:14:34.732 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:34.733+0000 7f93cdffb700 1 -- 172.21.15.79:0/3217018100 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f93c4059720 con 0x7f93d00ff5a0 2024-05-04T12:14:34.915 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:34.916+0000 7f93d6ce6700 1 -- 172.21.15.79:0/3217018100 --> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f93d0197220 con 0x7f93c006c4e0 2024-05-04T12:14:34.917 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:34.918+0000 7f93cdffb700 1 -- 172.21.15.79:0/3217018100 <== mgr.14219 v2:172.21.15.79:6800/402764217 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+28109 (secure 0 0 0) 0x7f93d0197220 con 0x7f93c006c4e0 2024-05-04T12:14:34.917 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-05-04T12:14:34.919 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:34.920+0000 7f93d6ce6700 1 -- 172.21.15.79:0/3217018100 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f93c006c4e0 msgr2=0x7f93c006e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:14:34.919 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:34.920+0000 7f93d6ce6700 1 --2- 172.21.15.79:0/3217018100 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f93c006c4e0 0x7f93c006e9a0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f93bc00b5c0 tx=0x7f93bc005960 comp rx=0 tx=0).stop 2024-05-04T12:14:34.919 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:34.920+0000 7f93d6ce6700 1 -- 172.21.15.79:0/3217018100 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f93d00ff5a0 msgr2=0x7f93d019c6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:14:34.920 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:34.920+0000 7f93d6ce6700 1 --2- 172.21.15.79:0/3217018100 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f93d00ff5a0 0x7f93d019c6f0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f93c400b4b0 tx=0x7f93c400b870 comp rx=0 tx=0).stop 2024-05-04T12:14:34.920 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:34.920+0000 7f93d6ce6700 1 -- 172.21.15.79:0/3217018100 shutdown_connections 2024-05-04T12:14:34.920 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:34.920+0000 7f93d6ce6700 1 --2- 172.21.15.79:0/3217018100 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f93c006c4e0 0x7f93c006e9a0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:34.920 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:34.920+0000 7f93d6ce6700 1 --2- 172.21.15.79:0/3217018100 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f93d00ff5a0 0x7f93d019c6f0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:34.920 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:34.920+0000 7f93d6ce6700 1 --2- 172.21.15.79:0/3217018100 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f93d00fff50 0x7f93d019cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:34.920 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:34.920+0000 7f93d6ce6700 1 -- 172.21.15.79:0/3217018100 >> 172.21.15.79:0/3217018100 conn(0x7f93d0074e30 msgr2=0x7f93d00fdf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:14:34.920 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:34.920+0000 7f93d6ce6700 1 -- 172.21.15.79:0/3217018100 shutdown_connections 2024-05-04T12:14:34.920 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:34.920+0000 7f93d6ce6700 1 -- 172.21.15.79:0/3217018100 wait complete. 2024-05-04T12:14:34.920 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-05-04T12:14:35.195 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:14:35 smithi079 ceph-mon[27222]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-04T12:14:35.526 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:14:35 smithi146 ceph-mon[35311]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-04T12:14:35.594 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":159,"stamp":"2024-05-04T12:14:33.633796+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":165588,"kb_used_data":3876,"kb_used_omap":0,"kb_used_meta":161664,"kb_avail":562305324,"statfs":{"total":575970213888,"available":575800651776,"internally_reserved":0,"allocated":3969024,"data_stored":2606850,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":165543936},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002245"},"pg_stats":[{"pgid":"1.0","version":"23'117","reported_seq":185,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-05-04T12:13:43.141641+0000","last_change":"2024-05-04T12:13:16.062334+0000","last_active":"2024-05-04T12:13:43.141641+0000","last_peered":"2024-05-04T12:13:43.141641+0000","last_clean":"2024-05-04T12:13:43.141641+0000","last_became_active":"2024-05-04T12:13:16.062048+0000","last_became_peered":"2024-05-04T12:13:16.062048+0000","last_unstale":"2024-05-04T12:13:43.141641+0000","last_undegraded":"2024-05-04T12:13:43.141641+0000","last_fullsized":"2024-05-04T12:13:43.141641+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T12:12:28.048271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T12:12:28.048271+0000","last_clean_scrub_stamp":"2024-05-04T12:12:28.048271+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T17:46:08.340937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":39,"seq":167503724556,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27308,"kb_used_data":356,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717844,"statfs":{"total":95995035648,"available":95967072256,"internally_reserved":0,"allocated":364544,"data_stored":139291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52100000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67200000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59699999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45100000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.435}]}]},{"osd":4,"up_from":34,"seq":146028888082,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27308,"kb_used_data":356,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717844,"statfs":{"total":95995035648,"available":95967072256,"internally_reserved":0,"allocated":364544,"data_stored":139291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat May 4 12:14:16 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.64000000000000001}]},{"osd":1,"last update":"Sat May 4 12:14:16 2024","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.58399999999999996}]},{"osd":2,"last update":"Sat May 4 12:14:16 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.66900000000000004}]},{"osd":3,"last update":"Sat May 4 12:14:16 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"last":0.55000000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.628}]}]},{"osd":3,"up_from":26,"seq":111669149720,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27888,"kb_used_data":936,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717264,"statfs":{"total":95995035648,"available":95966478336,"internally_reserved":0,"allocated":958464,"data_stored":729659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat May 4 12:13:50 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.52800000000000002}]},{"osd":1,"last update":"Sat May 4 12:13:50 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.441},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.42799999999999999}]},{"osd":2,"last update":"Sat May 4 12:13:50 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.46500000000000002}]},{"osd":4,"last update":"Sat May 4 12:14:16 2024","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.376,"5min":0.376,"15min":0.376},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"last":0.35899999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.318}]}]},{"osd":2,"up_from":19,"seq":81604378653,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27308,"kb_used_data":356,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717844,"statfs":{"total":95995035648,"available":95967072256,"internally_reserved":0,"allocated":364544,"data_stored":139291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat May 4 12:14:29 2024","interfaces":[{"interface":"back","average":{"1min":0.32300000000000001,"5min":0.33900000000000002,"15min":0.34100000000000003},"min":{"1min":0.219,"5min":0.186,"15min":0.186},"max":{"1min":0.48499999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.219},{"interface":"front","average":{"1min":0.33100000000000002,"5min":0.36399999999999999,"15min":0.37},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"last":0.23599999999999999}]},{"osd":1,"last update":"Sat May 4 12:14:29 2024","interfaces":[{"interface":"back","average":{"1min":0.34399999999999997,"5min":0.35799999999999998,"15min":0.35999999999999999},"min":{"1min":0.26400000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"last":0.29799999999999999},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.36599999999999999,"15min":0.36699999999999999},"min":{"1min":0.23799999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"last":0.254}]},{"osd":3,"last update":"Sat May 4 12:13:51 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.61099999999999999}]},{"osd":4,"last update":"Sat May 4 12:14:17 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.50700000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58099999999999996}]}]},{"osd":0,"up_from":9,"seq":38654705704,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27888,"kb_used_data":936,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717264,"statfs":{"total":95995035648,"available":95966478336,"internally_reserved":0,"allocated":958464,"data_stored":729659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat May 4 12:14:03 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.35599999999999998,"15min":0.36099999999999999},"min":{"1min":0.223,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.46600000000000003,"5min":0.748,"15min":0.748},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.34100000000000003,"5min":0.36499999999999999,"15min":0.36899999999999999},"min":{"1min":0.193,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.50900000000000001,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.39200000000000002}]},{"osd":2,"last update":"Sat May 4 12:13:27 2024","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"last":0.40500000000000003}]},{"osd":3,"last update":"Sat May 4 12:13:53 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.65300000000000002}]},{"osd":4,"last update":"Sat May 4 12:14:18 2024","interfaces":[{"interface":"back","average":{"1min":1.105,"5min":1.105,"15min":1.105},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":10.659000000000001,"5min":10.659000000000001,"15min":10.659000000000001},"last":0.63200000000000001},{"interface":"front","average":{"1min":1.1200000000000001,"5min":1.1200000000000001,"15min":1.1200000000000001},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":10.603,"5min":10.603,"15min":10.603},"last":0.621}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56299999999999994}]}]},{"osd":1,"up_from":14,"seq":60129542178,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27888,"kb_used_data":936,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717264,"statfs":{"total":95995035648,"available":95966478336,"internally_reserved":0,"allocated":958464,"data_stored":729659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat May 4 12:14:03 2024","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":0.313,"15min":0.309},"min":{"1min":0.20300000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"last":0.28100000000000003},{"interface":"front","average":{"1min":0.33100000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":0.47399999999999998,"5min":0.57499999999999996,"15min":0.57499999999999996},"last":0.28899999999999998}]},{"osd":2,"last update":"Sat May 4 12:14:27 2024","interfaces":[{"interface":"back","average":{"1min":0.29899999999999999,"5min":0.33600000000000002,"15min":0.34200000000000003},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.46400000000000002,"5min":0.53300000000000003,"15min":0.53300000000000003},"last":0.25},{"interface":"front","average":{"1min":0.30399999999999999,"5min":0.34799999999999998,"15min":0.35599999999999998},"min":{"1min":0.23599999999999999,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.45500000000000002,"5min":0.47699999999999998,"15min":0.47699999999999998},"last":0.30499999999999999}]},{"osd":3,"last update":"Sat May 4 12:13:52 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.34399999999999997}]},{"osd":4,"last update":"Sat May 4 12:14:21 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.40699999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42799999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-04T12:14:35.595 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid d9b985c4-0a0e-11ef-bc97-c7b262605968 -- ceph pg dump --format=json 2024-05-04T12:14:35.802 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/d9b985c4-0a0e-11ef-bc97-c7b262605968/mon.smithi079/config 2024-05-04T12:14:36.195 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:14:36 smithi079 ceph-mon[27222]: from='client.14426 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-04T12:14:36.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:14:36 smithi146 ceph-mon[35311]: from='client.14426 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-04T12:14:37.171 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:14:37 smithi079 ceph-mon[27222]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-04T12:14:37.388 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:37.389+0000 7f800b2e0700 1 -- 172.21.15.79:0/206560617 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8004101850 msgr2=0x7f80041058a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:14:37.388 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:37.389+0000 7f800b2e0700 1 --2- 172.21.15.79:0/206560617 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8004101850 0x7f80041058a0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f7ff4009910 tx=0x7f7ff4009c20 comp rx=0 tx=0).stop 2024-05-04T12:14:37.388 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:37.389+0000 7f800b2e0700 1 -- 172.21.15.79:0/206560617 shutdown_connections 2024-05-04T12:14:37.388 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:37.389+0000 7f800b2e0700 1 --2- 172.21.15.79:0/206560617 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8004101850 0x7f80041058a0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:37.389 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:37.389+0000 7f800b2e0700 1 --2- 172.21.15.79:0/206560617 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8004100ea0 0x7f8004101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:37.389 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:37.389+0000 7f800b2e0700 1 -- 172.21.15.79:0/206560617 >> 172.21.15.79:0/206560617 conn(0x7f80040fc750 msgr2=0x7f80040feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:14:37.389 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:37.389+0000 7f800b2e0700 1 -- 172.21.15.79:0/206560617 shutdown_connections 2024-05-04T12:14:37.389 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:37.389+0000 7f800b2e0700 1 -- 172.21.15.79:0/206560617 wait complete. 2024-05-04T12:14:37.391 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:37.390+0000 7f800b2e0700 1 Processor -- start 2024-05-04T12:14:37.391 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:37.390+0000 7f800b2e0700 1 -- start start 2024-05-04T12:14:37.391 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:37.390+0000 7f800b2e0700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8004100ea0 0x7f8004073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:14:37.391 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:37.390+0000 7f800b2e0700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8004101850 0x7f80040735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:14:37.391 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:37.390+0000 7f800b2e0700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8004073b10 con 0x7f8004100ea0 2024-05-04T12:14:37.391 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:37.390+0000 7f800b2e0700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8004073c50 con 0x7f8004101850 2024-05-04T12:14:37.392 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:37.390+0000 7f800907c700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8004100ea0 0x7f8004073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:14:37.392 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:37.390+0000 7f800907c700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8004100ea0 0x7f8004073090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:59224/0 (socket says 172.21.15.79:59224) 2024-05-04T12:14:37.392 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:37.390+0000 7f800907c700 1 -- 172.21.15.79:0/2254598445 learned_addr learned my addr 172.21.15.79:0/2254598445 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:14:37.392 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:37.390+0000 7f800887b700 1 --2- 172.21.15.79:0/2254598445 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8004101850 0x7f80040735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:14:37.392 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:37.390+0000 7f800907c700 1 -- 172.21.15.79:0/2254598445 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8004101850 msgr2=0x7f80040735d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:14:37.392 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:37.390+0000 7f800907c700 1 --2- 172.21.15.79:0/2254598445 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8004101850 0x7f80040735d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:37.392 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:37.390+0000 7f800907c700 1 -- 172.21.15.79:0/2254598445 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ff40095f0 con 0x7f8004100ea0 2024-05-04T12:14:37.392 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:37.390+0000 7f800907c700 1 --2- 172.21.15.79:0/2254598445 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8004100ea0 0x7f8004073090 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f800000d8d0 tx=0x7f800000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:14:37.392 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:37.391+0000 7f7ffa7fc700 1 -- 172.21.15.79:0/2254598445 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8000009940 con 0x7f8004100ea0 2024-05-04T12:14:37.392 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:37.391+0000 7f7ffa7fc700 1 -- 172.21.15.79:0/2254598445 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f8000010460 con 0x7f8004100ea0 2024-05-04T12:14:37.392 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:37.391+0000 7f800b2e0700 1 -- 172.21.15.79:0/2254598445 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80041a2b00 con 0x7f8004100ea0 2024-05-04T12:14:37.392 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:37.391+0000 7f7ffa7fc700 1 -- 172.21.15.79:0/2254598445 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f800000f5f0 con 0x7f8004100ea0 2024-05-04T12:14:37.392 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:37.391+0000 7f800b2e0700 1 -- 172.21.15.79:0/2254598445 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80041a2f10 con 0x7f8004100ea0 2024-05-04T12:14:37.393 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:37.392+0000 7f800b2e0700 1 -- 172.21.15.79:0/2254598445 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80041092a0 con 0x7f8004100ea0 2024-05-04T12:14:37.393 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:37.392+0000 7f7ffa7fc700 1 -- 172.21.15.79:0/2254598445 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f80000105d0 con 0x7f8004100ea0 2024-05-04T12:14:37.393 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:37.392+0000 7f7ffa7fc700 1 --2- 172.21.15.79:0/2254598445 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f7ff006c4e0 0x7f7ff006e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:14:37.393 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:37.393+0000 7f7ffa7fc700 1 -- 172.21.15.79:0/2254598445 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f800008b4c0 con 0x7f8004100ea0 2024-05-04T12:14:37.393 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:37.393+0000 7f800887b700 1 --2- 172.21.15.79:0/2254598445 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f7ff006c4e0 0x7f7ff006e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:14:37.394 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:37.393+0000 7f800887b700 1 --2- 172.21.15.79:0/2254598445 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f7ff006c4e0 0x7f7ff006e9a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f8004074880 tx=0x7f7ff401a040 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:14:37.396 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:37.397+0000 7f7ffa7fc700 1 -- 172.21.15.79:0/2254598445 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f8000055fe0 con 0x7f8004100ea0 2024-05-04T12:14:37.489 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:14:37 smithi146 ceph-mon[35311]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-04T12:14:37.579 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:37.580+0000 7f800b2e0700 1 -- 172.21.15.79:0/2254598445 --> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f80040611d0 con 0x7f7ff006c4e0 2024-05-04T12:14:37.581 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:37.582+0000 7f7ffa7fc700 1 -- 172.21.15.79:0/2254598445 <== mgr.14219 v2:172.21.15.79:6800/402764217 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+28038 (secure 0 0 0) 0x7f80040611d0 con 0x7f7ff006c4e0 2024-05-04T12:14:37.581 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-05-04T12:14:37.583 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:37.584+0000 7f800b2e0700 1 -- 172.21.15.79:0/2254598445 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f7ff006c4e0 msgr2=0x7f7ff006e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:14:37.583 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:37.584+0000 7f800b2e0700 1 --2- 172.21.15.79:0/2254598445 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f7ff006c4e0 0x7f7ff006e9a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f8004074880 tx=0x7f7ff401a040 comp rx=0 tx=0).stop 2024-05-04T12:14:37.583 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:37.584+0000 7f800b2e0700 1 -- 172.21.15.79:0/2254598445 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8004100ea0 msgr2=0x7f8004073090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:14:37.583 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:37.584+0000 7f800b2e0700 1 --2- 172.21.15.79:0/2254598445 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8004100ea0 0x7f8004073090 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f800000d8d0 tx=0x7f800000dc90 comp rx=0 tx=0).stop 2024-05-04T12:14:37.584 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:37.584+0000 7f800b2e0700 1 -- 172.21.15.79:0/2254598445 shutdown_connections 2024-05-04T12:14:37.584 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:37.584+0000 7f800b2e0700 1 --2- 172.21.15.79:0/2254598445 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f7ff006c4e0 0x7f7ff006e9a0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:37.584 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:37.584+0000 7f800b2e0700 1 --2- 172.21.15.79:0/2254598445 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8004100ea0 0x7f8004073090 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:37.584 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:37.584+0000 7f800b2e0700 1 --2- 172.21.15.79:0/2254598445 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8004101850 0x7f80040735d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:37.584 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:37.584+0000 7f800b2e0700 1 -- 172.21.15.79:0/2254598445 >> 172.21.15.79:0/2254598445 conn(0x7f80040fc750 msgr2=0x7f80040fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:14:37.584 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:37.584+0000 7f800b2e0700 1 -- 172.21.15.79:0/2254598445 shutdown_connections 2024-05-04T12:14:37.584 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:37.584+0000 7f800b2e0700 1 -- 172.21.15.79:0/2254598445 wait complete. 2024-05-04T12:14:37.584 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-05-04T12:14:38.129 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":160,"stamp":"2024-05-04T12:14:35.634155+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":165588,"kb_used_data":3876,"kb_used_omap":0,"kb_used_meta":161664,"kb_avail":562305324,"statfs":{"total":575970213888,"available":575800651776,"internally_reserved":0,"allocated":3969024,"data_stored":2606850,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":165543936},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002139"},"pg_stats":[{"pgid":"1.0","version":"23'117","reported_seq":185,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-05-04T12:13:43.141641+0000","last_change":"2024-05-04T12:13:16.062334+0000","last_active":"2024-05-04T12:13:43.141641+0000","last_peered":"2024-05-04T12:13:43.141641+0000","last_clean":"2024-05-04T12:13:43.141641+0000","last_became_active":"2024-05-04T12:13:16.062048+0000","last_became_peered":"2024-05-04T12:13:16.062048+0000","last_unstale":"2024-05-04T12:13:43.141641+0000","last_undegraded":"2024-05-04T12:13:43.141641+0000","last_fullsized":"2024-05-04T12:13:43.141641+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T12:12:28.048271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T12:12:28.048271+0000","last_clean_scrub_stamp":"2024-05-04T12:12:28.048271+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T17:46:08.340937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":39,"seq":167503724556,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27308,"kb_used_data":356,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717844,"statfs":{"total":95995035648,"available":95967072256,"internally_reserved":0,"allocated":364544,"data_stored":139291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52100000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67200000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59699999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45100000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.435}]}]},{"osd":4,"up_from":34,"seq":146028888082,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27308,"kb_used_data":356,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717844,"statfs":{"total":95995035648,"available":95967072256,"internally_reserved":0,"allocated":364544,"data_stored":139291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat May 4 12:14:16 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.64000000000000001}]},{"osd":1,"last update":"Sat May 4 12:14:16 2024","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.58399999999999996}]},{"osd":2,"last update":"Sat May 4 12:14:16 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.66900000000000004}]},{"osd":3,"last update":"Sat May 4 12:14:16 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"last":0.55000000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.628}]}]},{"osd":3,"up_from":26,"seq":111669149720,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27888,"kb_used_data":936,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717264,"statfs":{"total":95995035648,"available":95966478336,"internally_reserved":0,"allocated":958464,"data_stored":729659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat May 4 12:13:50 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.52800000000000002}]},{"osd":1,"last update":"Sat May 4 12:13:50 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.441},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.42799999999999999}]},{"osd":2,"last update":"Sat May 4 12:13:50 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.46500000000000002}]},{"osd":4,"last update":"Sat May 4 12:14:16 2024","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.376,"5min":0.376,"15min":0.376},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"last":0.35899999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.318}]}]},{"osd":2,"up_from":19,"seq":81604378653,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27308,"kb_used_data":356,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717844,"statfs":{"total":95995035648,"available":95967072256,"internally_reserved":0,"allocated":364544,"data_stored":139291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat May 4 12:14:29 2024","interfaces":[{"interface":"back","average":{"1min":0.32300000000000001,"5min":0.33900000000000002,"15min":0.34100000000000003},"min":{"1min":0.219,"5min":0.186,"15min":0.186},"max":{"1min":0.48499999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.219},{"interface":"front","average":{"1min":0.33100000000000002,"5min":0.36399999999999999,"15min":0.37},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"last":0.23599999999999999}]},{"osd":1,"last update":"Sat May 4 12:14:29 2024","interfaces":[{"interface":"back","average":{"1min":0.34399999999999997,"5min":0.35799999999999998,"15min":0.35999999999999999},"min":{"1min":0.26400000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"last":0.29799999999999999},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.36599999999999999,"15min":0.36699999999999999},"min":{"1min":0.23799999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"last":0.254}]},{"osd":3,"last update":"Sat May 4 12:13:51 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.61099999999999999}]},{"osd":4,"last update":"Sat May 4 12:14:17 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.50700000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58099999999999996}]}]},{"osd":0,"up_from":9,"seq":38654705705,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27888,"kb_used_data":936,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717264,"statfs":{"total":95995035648,"available":95966478336,"internally_reserved":0,"allocated":958464,"data_stored":729659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat May 4 12:14:03 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.35599999999999998,"15min":0.36099999999999999},"min":{"1min":0.223,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.46600000000000003,"5min":0.748,"15min":0.748},"last":0.37},{"interface":"front","average":{"1min":0.34100000000000003,"5min":0.36499999999999999,"15min":0.36899999999999999},"min":{"1min":0.193,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.50900000000000001,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.34200000000000003}]},{"osd":2,"last update":"Sat May 4 12:14:29 2024","interfaces":[{"interface":"back","average":{"1min":0.318,"5min":0.35399999999999998,"15min":0.35999999999999999},"min":{"1min":0.23000000000000001,"5min":0.219,"15min":0.219},"max":{"1min":0.46400000000000002,"5min":0.48599999999999999,"15min":0.48599999999999999},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.318,"5min":0.374,"15min":0.38400000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.40500000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"last":0.40300000000000002}]},{"osd":3,"last update":"Sat May 4 12:13:53 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.53400000000000003}]},{"osd":4,"last update":"Sat May 4 12:14:18 2024","interfaces":[{"interface":"back","average":{"1min":1.105,"5min":1.105,"15min":1.105},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":10.659000000000001,"5min":10.659000000000001,"15min":10.659000000000001},"last":0.55600000000000005},{"interface":"front","average":{"1min":1.1200000000000001,"5min":1.1200000000000001,"15min":1.1200000000000001},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":10.603,"5min":10.603,"15min":10.603},"last":0.54700000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57399999999999995}]}]},{"osd":1,"up_from":14,"seq":60129542178,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27888,"kb_used_data":936,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717264,"statfs":{"total":95995035648,"available":95966478336,"internally_reserved":0,"allocated":958464,"data_stored":729659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat May 4 12:14:03 2024","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":0.313,"15min":0.309},"min":{"1min":0.20300000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"last":0.28100000000000003},{"interface":"front","average":{"1min":0.33100000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":0.47399999999999998,"5min":0.57499999999999996,"15min":0.57499999999999996},"last":0.28899999999999998}]},{"osd":2,"last update":"Sat May 4 12:14:27 2024","interfaces":[{"interface":"back","average":{"1min":0.29899999999999999,"5min":0.33600000000000002,"15min":0.34200000000000003},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.46400000000000002,"5min":0.53300000000000003,"15min":0.53300000000000003},"last":0.25},{"interface":"front","average":{"1min":0.30399999999999999,"5min":0.34799999999999998,"15min":0.35599999999999998},"min":{"1min":0.23599999999999999,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.45500000000000002,"5min":0.47699999999999998,"15min":0.47699999999999998},"last":0.30499999999999999}]},{"osd":3,"last update":"Sat May 4 12:13:52 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.34399999999999997}]},{"osd":4,"last update":"Sat May 4 12:14:21 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.40699999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42799999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-04T12:14:38.130 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-05-04T12:14:38.130 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-05-04T12:14:38.130 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-05-04T12:14:38.130 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid d9b985c4-0a0e-11ef-bc97-c7b262605968 -- ceph health --format=json 2024-05-04T12:14:38.333 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/d9b985c4-0a0e-11ef-bc97-c7b262605968/mon.smithi079/config 2024-05-04T12:14:39.390 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:14:39 smithi079 ceph-mon[27222]: from='client.14430 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-04T12:14:39.390 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:14:39 smithi079 ceph-mon[27222]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-04T12:14:39.526 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:14:39 smithi146 ceph-mon[35311]: from='client.14430 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-04T12:14:39.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:14:39 smithi146 ceph-mon[35311]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-04T12:14:39.867 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:39.868+0000 7fd4f8e13700 1 -- 172.21.15.79:0/256582256 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd4f4100ea0 msgr2=0x7fd4f4101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:14:39.868 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:39.868+0000 7fd4f8e13700 1 --2- 172.21.15.79:0/256582256 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd4f4100ea0 0x7fd4f4101280 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fd4e8009910 tx=0x7fd4e8009c20 comp rx=0 tx=0).stop 2024-05-04T12:14:39.868 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:39.868+0000 7fd4f8e13700 1 -- 172.21.15.79:0/256582256 shutdown_connections 2024-05-04T12:14:39.868 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:39.868+0000 7fd4f8e13700 1 --2- 172.21.15.79:0/256582256 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd4f4101850 0x7fd4f41058a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:39.868 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:39.868+0000 7fd4f8e13700 1 --2- 172.21.15.79:0/256582256 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd4f4100ea0 0x7fd4f4101280 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:39.868 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:39.868+0000 7fd4f8e13700 1 -- 172.21.15.79:0/256582256 >> 172.21.15.79:0/256582256 conn(0x7fd4f40fc750 msgr2=0x7fd4f40feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:14:39.868 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:39.868+0000 7fd4f8e13700 1 -- 172.21.15.79:0/256582256 shutdown_connections 2024-05-04T12:14:39.868 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:39.868+0000 7fd4f8e13700 1 -- 172.21.15.79:0/256582256 wait complete. 2024-05-04T12:14:39.871 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:39.869+0000 7fd4f8e13700 1 Processor -- start 2024-05-04T12:14:39.871 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:39.869+0000 7fd4f8e13700 1 -- start start 2024-05-04T12:14:39.871 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:39.869+0000 7fd4f8e13700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd4f4100ea0 0x7fd4f419c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:14:39.871 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:39.869+0000 7fd4f8e13700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd4f4101850 0x7fd4f419ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:14:39.871 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:39.869+0000 7fd4f8e13700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4f419d350 con 0x7fd4f4101850 2024-05-04T12:14:39.871 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:39.869+0000 7fd4f8e13700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4f4196800 con 0x7fd4f4100ea0 2024-05-04T12:14:39.871 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:39.869+0000 7fd4f1d9b700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd4f4101850 0x7fd4f419ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:14:39.872 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:39.869+0000 7fd4f1d9b700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd4f4101850 0x7fd4f419ccc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:59250/0 (socket says 172.21.15.79:59250) 2024-05-04T12:14:39.872 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:39.869+0000 7fd4f1d9b700 1 -- 172.21.15.79:0/3036457272 learned_addr learned my addr 172.21.15.79:0/3036457272 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:14:39.872 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:39.870+0000 7fd4f259c700 1 --2- 172.21.15.79:0/3036457272 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd4f4100ea0 0x7fd4f419c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:14:39.872 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:39.870+0000 7fd4f1d9b700 1 -- 172.21.15.79:0/3036457272 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd4f4100ea0 msgr2=0x7fd4f419c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:14:39.872 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:39.870+0000 7fd4f1d9b700 1 --2- 172.21.15.79:0/3036457272 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd4f4100ea0 0x7fd4f419c780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:39.872 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:39.870+0000 7fd4f1d9b700 1 -- 172.21.15.79:0/3036457272 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4e80095f0 con 0x7fd4f4101850 2024-05-04T12:14:39.872 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:39.870+0000 7fd4f259c700 1 --2- 172.21.15.79:0/3036457272 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd4f4100ea0 0x7fd4f419c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-04T12:14:39.872 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:39.870+0000 7fd4f1d9b700 1 --2- 172.21.15.79:0/3036457272 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd4f4101850 0x7fd4f419ccc0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fd4e400b480 tx=0x7fd4e400b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:14:39.872 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:39.870+0000 7fd4e37fe700 1 -- 172.21.15.79:0/3036457272 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd4e40118c0 con 0x7fd4f4101850 2024-05-04T12:14:39.872 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:39.870+0000 7fd4f8e13700 1 -- 172.21.15.79:0/3036457272 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4f4196ae0 con 0x7fd4f4101850 2024-05-04T12:14:39.872 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:39.870+0000 7fd4e37fe700 1 -- 172.21.15.79:0/3036457272 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fd4e4010380 con 0x7fd4f4101850 2024-05-04T12:14:39.872 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:39.870+0000 7fd4e37fe700 1 -- 172.21.15.79:0/3036457272 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd4e400f4f0 con 0x7fd4f4101850 2024-05-04T12:14:39.873 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:39.870+0000 7fd4f8e13700 1 -- 172.21.15.79:0/3036457272 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4f4197030 con 0x7fd4f4101850 2024-05-04T12:14:39.873 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:39.871+0000 7fd4f8e13700 1 -- 172.21.15.79:0/3036457272 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4f4109240 con 0x7fd4f4101850 2024-05-04T12:14:39.873 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:39.871+0000 7fd4e37fe700 1 -- 172.21.15.79:0/3036457272 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7fd4e4011a20 con 0x7fd4f4101850 2024-05-04T12:14:39.873 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:39.872+0000 7fd4e37fe700 1 --2- 172.21.15.79:0/3036457272 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7fd4dc06c4e0 0x7fd4dc06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:14:39.873 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:39.872+0000 7fd4e37fe700 1 -- 172.21.15.79:0/3036457272 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fd4e408b1a0 con 0x7fd4f4101850 2024-05-04T12:14:39.873 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:39.872+0000 7fd4f259c700 1 --2- 172.21.15.79:0/3036457272 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7fd4dc06c4e0 0x7fd4dc06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:14:39.873 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:39.872+0000 7fd4f259c700 1 --2- 172.21.15.79:0/3036457272 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7fd4dc06c4e0 0x7fd4dc06e9a0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fd4e800b5c0 tx=0x7fd4e8005480 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:14:39.875 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:39.876+0000 7fd4e37fe700 1 -- 172.21.15.79:0/3036457272 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fd4e4059750 con 0x7fd4f4101850 2024-05-04T12:14:40.111 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:40.112+0000 7fd4f8e13700 1 -- 172.21.15.79:0/3036457272 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fd4f404f2e0 con 0x7fd4f4101850 2024-05-04T12:14:40.111 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:40.112+0000 7fd4e37fe700 1 -- 172.21.15.79:0/3036457272 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7fd4e40592e0 con 0x7fd4f4101850 2024-05-04T12:14:40.112 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-05-04T12:14:40.112 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-05-04T12:14:40.114 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:40.114+0000 7fd4f8e13700 1 -- 172.21.15.79:0/3036457272 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7fd4dc06c4e0 msgr2=0x7fd4dc06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:14:40.114 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:40.114+0000 7fd4f8e13700 1 --2- 172.21.15.79:0/3036457272 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7fd4dc06c4e0 0x7fd4dc06e9a0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fd4e800b5c0 tx=0x7fd4e8005480 comp rx=0 tx=0).stop 2024-05-04T12:14:40.114 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:40.114+0000 7fd4f8e13700 1 -- 172.21.15.79:0/3036457272 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd4f4101850 msgr2=0x7fd4f419ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:14:40.114 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:40.114+0000 7fd4f8e13700 1 --2- 172.21.15.79:0/3036457272 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd4f4101850 0x7fd4f419ccc0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fd4e400b480 tx=0x7fd4e400b840 comp rx=0 tx=0).stop 2024-05-04T12:14:40.114 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:40.114+0000 7fd4f8e13700 1 -- 172.21.15.79:0/3036457272 shutdown_connections 2024-05-04T12:14:40.114 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:40.114+0000 7fd4f8e13700 1 --2- 172.21.15.79:0/3036457272 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7fd4dc06c4e0 0x7fd4dc06e9a0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:40.114 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:40.114+0000 7fd4f8e13700 1 --2- 172.21.15.79:0/3036457272 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd4f4100ea0 0x7fd4f419c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:40.114 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:40.114+0000 7fd4f8e13700 1 --2- 172.21.15.79:0/3036457272 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd4f4101850 0x7fd4f419ccc0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:40.114 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:40.114+0000 7fd4f8e13700 1 -- 172.21.15.79:0/3036457272 >> 172.21.15.79:0/3036457272 conn(0x7fd4f40fc750 msgr2=0x7fd4f40fde10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:14:40.114 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:40.114+0000 7fd4f8e13700 1 -- 172.21.15.79:0/3036457272 shutdown_connections 2024-05-04T12:14:40.115 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:40.114+0000 7fd4f8e13700 1 -- 172.21.15.79:0/3036457272 wait complete. 2024-05-04T12:14:40.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:14:40 smithi079 ceph-mon[27222]: from='client.? 172.21.15.79:0/3036457272' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-05-04T12:14:40.526 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:14:40 smithi146 ceph-mon[35311]: from='client.? 172.21.15.79:0/3036457272' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-05-04T12:14:40.779 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-05-04T12:14:40.779 INFO:tasks.cephadm:Setup complete, yielding 2024-05-04T12:14:40.780 INFO:teuthology.run_tasks:Running task print... 2024-05-04T12:14:40.789 INFO:teuthology.task.print:**** done end installing v18.2.1 cephadm ... 2024-05-04T12:14:40.789 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-05-04T12:14:40.798 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi079.front.sepia.ceph.com 2024-05-04T12:14:40.798 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d9b985c4-0a0e-11ef-bc97-c7b262605968 -- bash -c 'ceph config set mgr mgr/cephadm/use_repo_digest true --force' 2024-05-04T12:14:41.003 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/d9b985c4-0a0e-11ef-bc97-c7b262605968/mon.smithi079/config 2024-05-04T12:14:41.428 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:14:41 smithi146 ceph-mon[35311]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-04T12:14:41.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:14:41 smithi079 ceph-mon[27222]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-04T12:14:42.422 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:14:42 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-04T12:14:42.526 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:14:42 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-04T12:14:42.596 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:42.596+0000 7fa63d3fa700 1 -- 172.21.15.79:0/291520163 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa6380ffbc0 msgr2=0x7fa6380fffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:14:42.596 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:42.596+0000 7fa63d3fa700 1 --2- 172.21.15.79:0/291520163 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa6380ffbc0 0x7fa6380fffe0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fa630009910 tx=0x7fa630009c20 comp rx=0 tx=0).stop 2024-05-04T12:14:42.596 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:42.596+0000 7fa63d3fa700 1 -- 172.21.15.79:0/291520163 shutdown_connections 2024-05-04T12:14:42.596 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:42.596+0000 7fa63d3fa700 1 --2- 172.21.15.79:0/291520163 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa638100dc0 0x7fa638101220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:42.596 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:42.596+0000 7fa63d3fa700 1 --2- 172.21.15.79:0/291520163 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa6380ffbc0 0x7fa6380fffe0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:42.596 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:42.596+0000 7fa63d3fa700 1 -- 172.21.15.79:0/291520163 >> 172.21.15.79:0/291520163 conn(0x7fa6380fb160 msgr2=0x7fa6380fd5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:14:42.596 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:42.596+0000 7fa63d3fa700 1 -- 172.21.15.79:0/291520163 shutdown_connections 2024-05-04T12:14:42.596 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:42.596+0000 7fa63d3fa700 1 -- 172.21.15.79:0/291520163 wait complete. 2024-05-04T12:14:42.596 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:42.597+0000 7fa63d3fa700 1 Processor -- start 2024-05-04T12:14:42.596 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:42.597+0000 7fa63d3fa700 1 -- start start 2024-05-04T12:14:42.598 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:42.597+0000 7fa63d3fa700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa6380ffbc0 0x7fa6381967f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:14:42.598 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:42.597+0000 7fa63d3fa700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa638100dc0 0x7fa638196d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:14:42.598 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:42.597+0000 7fa63d3fa700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa638197350 con 0x7fa638100dc0 2024-05-04T12:14:42.598 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:42.597+0000 7fa63d3fa700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa638197490 con 0x7fa6380ffbc0 2024-05-04T12:14:42.598 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:42.597+0000 7fa6367fc700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa638100dc0 0x7fa638196d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:14:42.598 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:42.597+0000 7fa6367fc700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa638100dc0 0x7fa638196d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:59278/0 (socket says 172.21.15.79:59278) 2024-05-04T12:14:42.598 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:42.597+0000 7fa6367fc700 1 -- 172.21.15.79:0/3282417619 learned_addr learned my addr 172.21.15.79:0/3282417619 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:14:42.599 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:42.597+0000 7fa636ffd700 1 --2- 172.21.15.79:0/3282417619 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa6380ffbc0 0x7fa6381967f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:14:42.599 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:42.598+0000 7fa6367fc700 1 -- 172.21.15.79:0/3282417619 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa6380ffbc0 msgr2=0x7fa6381967f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:14:42.599 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:42.598+0000 7fa6367fc700 1 --2- 172.21.15.79:0/3282417619 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa6380ffbc0 0x7fa6381967f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:42.599 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:42.598+0000 7fa6367fc700 1 -- 172.21.15.79:0/3282417619 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6300095f0 con 0x7fa638100dc0 2024-05-04T12:14:42.599 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:42.598+0000 7fa636ffd700 1 --2- 172.21.15.79:0/3282417619 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa6380ffbc0 0x7fa6381967f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-04T12:14:42.599 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:42.598+0000 7fa6367fc700 1 --2- 172.21.15.79:0/3282417619 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa638100dc0 0x7fa638196d30 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fa62000e920 tx=0x7fa62000ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:14:42.599 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:42.598+0000 7fa62ffff700 1 -- 172.21.15.79:0/3282417619 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa62000c9d0 con 0x7fa638100dc0 2024-05-04T12:14:42.599 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:42.598+0000 7fa62ffff700 1 -- 172.21.15.79:0/3282417619 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fa620010430 con 0x7fa638100dc0 2024-05-04T12:14:42.599 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:42.598+0000 7fa63d3fa700 1 -- 172.21.15.79:0/3282417619 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa63819bf40 con 0x7fa638100dc0 2024-05-04T12:14:42.599 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:42.598+0000 7fa62ffff700 1 -- 172.21.15.79:0/3282417619 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa620018650 con 0x7fa638100dc0 2024-05-04T12:14:42.599 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:42.598+0000 7fa63d3fa700 1 -- 172.21.15.79:0/3282417619 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa63819c460 con 0x7fa638100dc0 2024-05-04T12:14:42.599 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:42.599+0000 7fa63d3fa700 1 -- 172.21.15.79:0/3282417619 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa638068af0 con 0x7fa638100dc0 2024-05-04T12:14:42.599 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:42.599+0000 7fa62ffff700 1 -- 172.21.15.79:0/3282417619 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7fa6200105a0 con 0x7fa638100dc0 2024-05-04T12:14:42.600 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:42.600+0000 7fa62ffff700 1 --2- 172.21.15.79:0/3282417619 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7fa62406c4e0 0x7fa62406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:14:42.600 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:42.600+0000 7fa62ffff700 1 -- 172.21.15.79:0/3282417619 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fa620014070 con 0x7fa638100dc0 2024-05-04T12:14:42.600 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:42.600+0000 7fa636ffd700 1 --2- 172.21.15.79:0/3282417619 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7fa62406c4e0 0x7fa62406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:14:42.600 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:42.600+0000 7fa636ffd700 1 --2- 172.21.15.79:0/3282417619 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7fa62406c4e0 0x7fa62406e9a0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fa63000b6f0 tx=0x7fa630005940 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:14:42.603 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:42.604+0000 7fa62ffff700 1 -- 172.21.15.79:0/3282417619 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fa6200520e0 con 0x7fa638100dc0 2024-05-04T12:14:42.795 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:42.796+0000 7fa63d3fa700 1 -- 172.21.15.79:0/3282417619 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/use_repo_digest}] v 0) v1 -- 0x7fa63819cb20 con 0x7fa638100dc0 2024-05-04T12:14:42.800 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:42.801+0000 7fa62ffff700 1 -- 172.21.15.79:0/3282417619 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/use_repo_digest}]=0 v20)=0 v20) v1 ==== 143+0+0 (secure 0 0 0) 0x7fa620059d60 con 0x7fa638100dc0 2024-05-04T12:14:42.802 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:42.803+0000 7fa63d3fa700 1 -- 172.21.15.79:0/3282417619 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7fa62406c4e0 msgr2=0x7fa62406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:14:42.802 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:42.803+0000 7fa63d3fa700 1 --2- 172.21.15.79:0/3282417619 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7fa62406c4e0 0x7fa62406e9a0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fa63000b6f0 tx=0x7fa630005940 comp rx=0 tx=0).stop 2024-05-04T12:14:42.802 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:42.803+0000 7fa63d3fa700 1 -- 172.21.15.79:0/3282417619 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa638100dc0 msgr2=0x7fa638196d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:14:42.803 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:42.803+0000 7fa63d3fa700 1 --2- 172.21.15.79:0/3282417619 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa638100dc0 0x7fa638196d30 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fa62000e920 tx=0x7fa62000ec30 comp rx=0 tx=0).stop 2024-05-04T12:14:42.803 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:42.803+0000 7fa63d3fa700 1 -- 172.21.15.79:0/3282417619 shutdown_connections 2024-05-04T12:14:42.803 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:42.803+0000 7fa63d3fa700 1 --2- 172.21.15.79:0/3282417619 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7fa62406c4e0 0x7fa62406e9a0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:42.803 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:42.803+0000 7fa63d3fa700 1 --2- 172.21.15.79:0/3282417619 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa6380ffbc0 0x7fa6381967f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:42.803 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:42.803+0000 7fa63d3fa700 1 --2- 172.21.15.79:0/3282417619 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa638100dc0 0x7fa638196d30 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:42.803 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:42.803+0000 7fa63d3fa700 1 -- 172.21.15.79:0/3282417619 >> 172.21.15.79:0/3282417619 conn(0x7fa6380fb160 msgr2=0x7fa638068e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:14:42.803 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:42.803+0000 7fa63d3fa700 1 -- 172.21.15.79:0/3282417619 shutdown_connections 2024-05-04T12:14:42.803 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:42.803+0000 7fa63d3fa700 1 -- 172.21.15.79:0/3282417619 wait complete. 2024-05-04T12:14:43.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:14:43 smithi079 ceph-mon[27222]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-04T12:14:43.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:14:43 smithi079 ceph-mon[27222]: from='client.? 172.21.15.79:0/3282417619' entity='client.admin' 2024-05-04T12:14:43.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:14:43 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-04T12:14:43.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:14:43 smithi146 ceph-mon[35311]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-04T12:14:43.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:14:43 smithi146 ceph-mon[35311]: from='client.? 172.21.15.79:0/3282417619' entity='client.admin' 2024-05-04T12:14:43.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:14:43 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-04T12:14:43.606 INFO:teuthology.run_tasks:Running task print... 2024-05-04T12:14:43.615 INFO:teuthology.task.print:**** done cephadm.shell ceph config set mgr... 2024-05-04T12:14:43.615 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-05-04T12:14:43.637 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi079.front.sepia.ceph.com 2024-05-04T12:14:43.637 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d9b985c4-0a0e-11ef-bc97-c7b262605968 -- bash -c 'ceph orch status' 2024-05-04T12:14:43.846 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/d9b985c4-0a0e-11ef-bc97-c7b262605968/mon.smithi079/config 2024-05-04T12:14:44.272 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:14:44 smithi079 ceph-mon[27222]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-04T12:14:44.272 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:14:44 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-04T12:14:44.272 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:14:44 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-04T12:14:44.272 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:14:44 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:14:44.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:14:44 smithi146 ceph-mon[35311]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-04T12:14:44.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:14:44 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-04T12:14:44.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:14:44 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-04T12:14:44.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:14:44 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:14:45.295 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:45.288+0000 7f3e1e44a700 1 -- 172.21.15.79:0/3486407256 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3e18100e10 msgr2=0x7f3e18101270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:14:45.295 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:45.288+0000 7f3e1e44a700 1 --2- 172.21.15.79:0/3486407256 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3e18100e10 0x7f3e18101270 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f3e08009910 tx=0x7f3e08009c20 comp rx=0 tx=0).stop 2024-05-04T12:14:45.295 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:45.288+0000 7f3e1e44a700 1 -- 172.21.15.79:0/3486407256 shutdown_connections 2024-05-04T12:14:45.295 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:45.288+0000 7f3e1e44a700 1 --2- 172.21.15.79:0/3486407256 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3e18100e10 0x7f3e18101270 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:45.295 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:45.288+0000 7f3e1e44a700 1 --2- 172.21.15.79:0/3486407256 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3e180ffc10 0x7f3e18100030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:45.295 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:45.288+0000 7f3e1e44a700 1 -- 172.21.15.79:0/3486407256 >> 172.21.15.79:0/3486407256 conn(0x7f3e180fb190 msgr2=0x7f3e180fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:14:45.295 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:45.288+0000 7f3e1e44a700 1 -- 172.21.15.79:0/3486407256 shutdown_connections 2024-05-04T12:14:45.295 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:45.288+0000 7f3e1e44a700 1 -- 172.21.15.79:0/3486407256 wait complete. 2024-05-04T12:14:45.295 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:45.289+0000 7f3e1e44a700 1 Processor -- start 2024-05-04T12:14:45.295 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:45.289+0000 7f3e1e44a700 1 -- start start 2024-05-04T12:14:45.295 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:45.289+0000 7f3e1e44a700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3e180ffc10 0x7f3e181965e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:14:45.295 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:45.289+0000 7f3e1e44a700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3e18100e10 0x7f3e18196b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:14:45.296 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:45.289+0000 7f3e1e44a700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e18197140 con 0x7f3e18100e10 2024-05-04T12:14:45.296 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:45.289+0000 7f3e1e44a700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e18197280 con 0x7f3e180ffc10 2024-05-04T12:14:45.296 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:45.289+0000 7f3e177fe700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3e18100e10 0x7f3e18196b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:14:45.296 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:45.289+0000 7f3e177fe700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3e18100e10 0x7f3e18196b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:32988/0 (socket says 172.21.15.79:32988) 2024-05-04T12:14:45.296 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:45.289+0000 7f3e177fe700 1 -- 172.21.15.79:0/3519514953 learned_addr learned my addr 172.21.15.79:0/3519514953 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:14:45.296 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:45.289+0000 7f3e17fff700 1 --2- 172.21.15.79:0/3519514953 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3e180ffc10 0x7f3e181965e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:14:45.296 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:45.290+0000 7f3e177fe700 1 -- 172.21.15.79:0/3519514953 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3e180ffc10 msgr2=0x7f3e181965e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:14:45.296 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:45.290+0000 7f3e177fe700 1 --2- 172.21.15.79:0/3519514953 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3e180ffc10 0x7f3e181965e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:45.296 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:45.290+0000 7f3e177fe700 1 -- 172.21.15.79:0/3519514953 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e080095f0 con 0x7f3e18100e10 2024-05-04T12:14:45.296 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:45.290+0000 7f3e177fe700 1 --2- 172.21.15.79:0/3519514953 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3e18100e10 0x7f3e18196b20 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f3e080092e0 tx=0x7f3e0800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:14:45.296 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:45.290+0000 7f3e157fa700 1 -- 172.21.15.79:0/3519514953 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3e0801d070 con 0x7f3e18100e10 2024-05-04T12:14:45.296 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:45.290+0000 7f3e1e44a700 1 -- 172.21.15.79:0/3519514953 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e1819bcd0 con 0x7f3e18100e10 2024-05-04T12:14:45.296 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:45.290+0000 7f3e157fa700 1 -- 172.21.15.79:0/3519514953 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f3e0800f460 con 0x7f3e18100e10 2024-05-04T12:14:45.296 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:45.290+0000 7f3e157fa700 1 -- 172.21.15.79:0/3519514953 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3e08021640 con 0x7f3e18100e10 2024-05-04T12:14:45.297 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:45.290+0000 7f3e1e44a700 1 -- 172.21.15.79:0/3519514953 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e1819c1c0 con 0x7f3e18100e10 2024-05-04T12:14:45.297 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:45.291+0000 7f3e1e44a700 1 -- 172.21.15.79:0/3519514953 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e181098a0 con 0x7f3e18100e10 2024-05-04T12:14:45.297 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:45.291+0000 7f3e157fa700 1 -- 172.21.15.79:0/3519514953 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f3e0800fa90 con 0x7f3e18100e10 2024-05-04T12:14:45.297 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:45.292+0000 7f3e157fa700 1 --2- 172.21.15.79:0/3519514953 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f3e0006c4e0 0x7f3e0006e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:14:45.297 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:45.292+0000 7f3e157fa700 1 -- 172.21.15.79:0/3519514953 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f3e0808c850 con 0x7f3e18100e10 2024-05-04T12:14:45.297 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:45.292+0000 7f3e17fff700 1 --2- 172.21.15.79:0/3519514953 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f3e0006c4e0 0x7f3e0006e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:14:45.297 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:45.292+0000 7f3e17fff700 1 --2- 172.21.15.79:0/3519514953 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f3e0006c4e0 0x7f3e0006e9a0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f3e0c006fd0 tx=0x7f3e0c008040 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:14:45.297 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:45.296+0000 7f3e157fa700 1 -- 172.21.15.79:0/3519514953 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f3e0805ae00 con 0x7f3e18100e10 2024-05-04T12:14:45.502 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:45.503+0000 7f3e1e44a700 1 -- 172.21.15.79:0/3519514953 --> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) v1 -- 0x7f3e18105760 con 0x7f3e0006c4e0 2024-05-04T12:14:45.503 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:45.504+0000 7f3e157fa700 1 -- 172.21.15.79:0/3519514953 <== mgr.14219 v2:172.21.15.79:6800/402764217 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f3e18105760 con 0x7f3e0006c4e0 2024-05-04T12:14:45.503 INFO:teuthology.orchestra.run.smithi079.stdout:Backend: cephadm 2024-05-04T12:14:45.503 INFO:teuthology.orchestra.run.smithi079.stdout:Available: Yes 2024-05-04T12:14:45.503 INFO:teuthology.orchestra.run.smithi079.stdout:Paused: No 2024-05-04T12:14:45.505 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:45.506+0000 7f3e1e44a700 1 -- 172.21.15.79:0/3519514953 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f3e0006c4e0 msgr2=0x7f3e0006e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:14:45.505 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:45.506+0000 7f3e1e44a700 1 --2- 172.21.15.79:0/3519514953 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f3e0006c4e0 0x7f3e0006e9a0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f3e0c006fd0 tx=0x7f3e0c008040 comp rx=0 tx=0).stop 2024-05-04T12:14:45.505 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:45.506+0000 7f3e1e44a700 1 -- 172.21.15.79:0/3519514953 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3e18100e10 msgr2=0x7f3e18196b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:14:45.505 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:45.506+0000 7f3e1e44a700 1 --2- 172.21.15.79:0/3519514953 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3e18100e10 0x7f3e18196b20 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f3e080092e0 tx=0x7f3e0800be20 comp rx=0 tx=0).stop 2024-05-04T12:14:45.506 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:45.506+0000 7f3e1e44a700 1 -- 172.21.15.79:0/3519514953 shutdown_connections 2024-05-04T12:14:45.506 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:45.506+0000 7f3e1e44a700 1 --2- 172.21.15.79:0/3519514953 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f3e0006c4e0 0x7f3e0006e9a0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:45.506 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:45.506+0000 7f3e1e44a700 1 --2- 172.21.15.79:0/3519514953 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3e180ffc10 0x7f3e181965e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:45.506 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:45.506+0000 7f3e1e44a700 1 --2- 172.21.15.79:0/3519514953 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3e18100e10 0x7f3e18196b20 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:45.506 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:45.506+0000 7f3e1e44a700 1 -- 172.21.15.79:0/3519514953 >> 172.21.15.79:0/3519514953 conn(0x7f3e180fb190 msgr2=0x7f3e18104040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:14:45.506 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:45.506+0000 7f3e1e44a700 1 -- 172.21.15.79:0/3519514953 shutdown_connections 2024-05-04T12:14:45.506 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:45.506+0000 7f3e1e44a700 1 -- 172.21.15.79:0/3519514953 wait complete. 2024-05-04T12:14:46.086 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d9b985c4-0a0e-11ef-bc97-c7b262605968 -- bash -c 'ceph orch ps' 2024-05-04T12:14:46.304 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/d9b985c4-0a0e-11ef-bc97-c7b262605968/mon.smithi079/config 2024-05-04T12:14:47.314 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:14:47 smithi079 ceph-mon[27222]: from='client.14442 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-04T12:14:47.314 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:14:47 smithi079 ceph-mon[27222]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-04T12:14:47.526 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:14:47 smithi146 ceph-mon[35311]: from='client.14442 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-04T12:14:47.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:14:47 smithi146 ceph-mon[35311]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-04T12:14:47.819 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:47.819+0000 7f24802da700 1 -- 172.21.15.79:0/2977501960 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2478101eb0 msgr2=0x7f2478102310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:14:47.819 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:47.819+0000 7f24802da700 1 --2- 172.21.15.79:0/2977501960 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2478101eb0 0x7f2478102310 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f246c009910 tx=0x7f246c009c20 comp rx=0 tx=0).stop 2024-05-04T12:14:47.819 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:47.819+0000 7f24802da700 1 -- 172.21.15.79:0/2977501960 shutdown_connections 2024-05-04T12:14:47.820 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:47.819+0000 7f24802da700 1 --2- 172.21.15.79:0/2977501960 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2478101eb0 0x7f2478102310 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:47.820 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:47.819+0000 7f24802da700 1 --2- 172.21.15.79:0/2977501960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2478100cb0 0x7f24781010d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:47.820 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:47.819+0000 7f24802da700 1 -- 172.21.15.79:0/2977501960 >> 172.21.15.79:0/2977501960 conn(0x7f24780fc230 msgr2=0x7f24780fe690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:14:47.820 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:47.819+0000 7f24802da700 1 -- 172.21.15.79:0/2977501960 shutdown_connections 2024-05-04T12:14:47.820 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:47.819+0000 7f24802da700 1 -- 172.21.15.79:0/2977501960 wait complete. 2024-05-04T12:14:47.820 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:47.820+0000 7f24802da700 1 Processor -- start 2024-05-04T12:14:47.820 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:47.820+0000 7f24802da700 1 -- start start 2024-05-04T12:14:47.820 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:47.820+0000 7f24802da700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2478100cb0 0x7f24781964b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:14:47.820 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:47.820+0000 7f24802da700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2478101eb0 0x7f24781969f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:14:47.820 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:47.820+0000 7f24802da700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2478197010 con 0x7f2478101eb0 2024-05-04T12:14:47.820 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:47.820+0000 7f24802da700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2478197150 con 0x7f2478100cb0 2024-05-04T12:14:47.820 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:47.820+0000 7f247d875700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2478101eb0 0x7f24781969f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:14:47.821 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:47.820+0000 7f247d875700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2478101eb0 0x7f24781969f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:33012/0 (socket says 172.21.15.79:33012) 2024-05-04T12:14:47.821 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:47.820+0000 7f247d875700 1 -- 172.21.15.79:0/3720424769 learned_addr learned my addr 172.21.15.79:0/3720424769 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:14:47.821 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:47.820+0000 7f247e076700 1 --2- 172.21.15.79:0/3720424769 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2478100cb0 0x7f24781964b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:14:47.821 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:47.820+0000 7f247d875700 1 -- 172.21.15.79:0/3720424769 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2478100cb0 msgr2=0x7f24781964b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:14:47.821 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:47.820+0000 7f247d875700 1 --2- 172.21.15.79:0/3720424769 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2478100cb0 0x7f24781964b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:47.821 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:47.820+0000 7f247d875700 1 -- 172.21.15.79:0/3720424769 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f246c0095f0 con 0x7f2478101eb0 2024-05-04T12:14:47.821 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:47.821+0000 7f247d875700 1 --2- 172.21.15.79:0/3720424769 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2478101eb0 0x7f24781969f0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f246c006010 tx=0x7f246c00bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:14:47.821 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:47.821+0000 7f24777fe700 1 -- 172.21.15.79:0/3720424769 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f246c01d070 con 0x7f2478101eb0 2024-05-04T12:14:47.821 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:47.821+0000 7f24802da700 1 -- 172.21.15.79:0/3720424769 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f247819bba0 con 0x7f2478101eb0 2024-05-04T12:14:47.821 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:47.821+0000 7f24777fe700 1 -- 172.21.15.79:0/3720424769 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f246c00f460 con 0x7f2478101eb0 2024-05-04T12:14:47.821 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:47.821+0000 7f24777fe700 1 -- 172.21.15.79:0/3720424769 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f246c021640 con 0x7f2478101eb0 2024-05-04T12:14:47.821 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:47.821+0000 7f24802da700 1 -- 172.21.15.79:0/3720424769 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f247819c090 con 0x7f2478101eb0 2024-05-04T12:14:47.822 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:47.822+0000 7f24802da700 1 -- 172.21.15.79:0/3720424769 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2478109810 con 0x7f2478101eb0 2024-05-04T12:14:47.822 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:47.822+0000 7f24777fe700 1 -- 172.21.15.79:0/3720424769 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f246c00fab0 con 0x7f2478101eb0 2024-05-04T12:14:47.823 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:47.822+0000 7f24777fe700 1 --2- 172.21.15.79:0/3720424769 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f246006c4e0 0x7f246006e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:14:47.823 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:47.823+0000 7f24777fe700 1 -- 172.21.15.79:0/3720424769 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f246c08c820 con 0x7f2478101eb0 2024-05-04T12:14:47.823 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:47.823+0000 7f247e076700 1 --2- 172.21.15.79:0/3720424769 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f246006c4e0 0x7f246006e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:14:47.823 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:47.823+0000 7f247e076700 1 --2- 172.21.15.79:0/3720424769 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f246006c4e0 0x7f246006e9a0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f2478101d10 tx=0x7f247000a380 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:14:47.825 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:47.827+0000 7f24777fe700 1 -- 172.21.15.79:0/3720424769 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f246c05add0 con 0x7f2478101eb0 2024-05-04T12:14:48.022 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:48.024+0000 7f24802da700 1 -- 172.21.15.79:0/3720424769 --> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f2478106800 con 0x7f246006c4e0 2024-05-04T12:14:48.034 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:48.033+0000 7f24777fe700 1 -- 172.21.15.79:0/3720424769 <== mgr.14219 v2:172.21.15.79:6800/402764217 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+2820 (secure 0 0 0) 0x7f2478106800 con 0x7f246006c4e0 2024-05-04T12:14:48.034 INFO:teuthology.orchestra.run.smithi079.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-05-04T12:14:48.035 INFO:teuthology.orchestra.run.smithi079.stdout:alertmanager.smithi079 smithi079 *:9093,9094 running (3m) 2m ago 5m 26.2M - 0.25.0 c8568f914cd2 65445b7ae71b 2024-05-04T12:14:48.035 INFO:teuthology.orchestra.run.smithi079.stdout:ceph-exporter.smithi079 smithi079 running (5m) 2m ago 5m 7025k - 18.2.1 5be31c24972a f691c99a7c23 2024-05-04T12:14:48.035 INFO:teuthology.orchestra.run.smithi079.stdout:ceph-exporter.smithi146 smithi146 running (4m) 66s ago 4m 6966k - 18.2.1 5be31c24972a 71288aa0ec98 2024-05-04T12:14:48.035 INFO:teuthology.orchestra.run.smithi079.stdout:crash.smithi079 smithi079 running (5m) 2m ago 5m 6656k - 18.2.1 5be31c24972a 6ce9cc3a3e35 2024-05-04T12:14:48.035 INFO:teuthology.orchestra.run.smithi079.stdout:crash.smithi146 smithi146 running (4m) 66s ago 4m 6660k - 18.2.1 5be31c24972a c42b91d1d7da 2024-05-04T12:14:48.035 INFO:teuthology.orchestra.run.smithi079.stdout:grafana.smithi079 smithi079 *:3000 running (3m) 2m ago 5m 81.1M - 9.4.7 954c08fa6188 a6fce20482ec 2024-05-04T12:14:48.035 INFO:teuthology.orchestra.run.smithi079.stdout:mgr.smithi079.dgcpwl smithi079 *:9283,8765,8443 running (6m) 2m ago 6m 481M - 18.2.1 5be31c24972a 21dc78f7be3a 2024-05-04T12:14:48.035 INFO:teuthology.orchestra.run.smithi079.stdout:mgr.smithi146.eubqih smithi146 *:8443,9283,8765 running (4m) 66s ago 4m 439M - 18.2.1 5be31c24972a e03895ea0bc8 2024-05-04T12:14:48.035 INFO:teuthology.orchestra.run.smithi079.stdout:mon.smithi079 smithi079 running (6m) 2m ago 6m 44.5M 2048M 18.2.1 5be31c24972a ce54f11f45df 2024-05-04T12:14:48.035 INFO:teuthology.orchestra.run.smithi079.stdout:mon.smithi146 smithi146 running (4m) 66s ago 4m 41.5M 2048M 18.2.1 5be31c24972a 085b6b3257ee 2024-05-04T12:14:48.035 INFO:teuthology.orchestra.run.smithi079.stdout:node-exporter.smithi079 smithi079 *:9100 running (5m) 2m ago 5m 16.3M - 1.5.0 0da6a335fe13 3502f0e86c17 2024-05-04T12:14:48.035 INFO:teuthology.orchestra.run.smithi079.stdout:node-exporter.smithi146 smithi146 *:9100 running (4m) 66s ago 4m 16.7M - 1.5.0 0da6a335fe13 19d4f0087b1f 2024-05-04T12:14:48.035 INFO:teuthology.orchestra.run.smithi079.stdout:osd.0 smithi079 running (3m) 2m ago 3m 44.4M 3946M 18.2.1 5be31c24972a ee518ff3d765 2024-05-04T12:14:48.036 INFO:teuthology.orchestra.run.smithi079.stdout:osd.1 smithi079 running (2m) 2m ago 2m 39.3M 3946M 18.2.1 5be31c24972a 57cfd8dc67dd 2024-05-04T12:14:48.036 INFO:teuthology.orchestra.run.smithi079.stdout:osd.2 smithi079 running (2m) 2m ago 2m 39.0M 3946M 18.2.1 5be31c24972a 4778a780462b 2024-05-04T12:14:48.036 INFO:teuthology.orchestra.run.smithi079.stdout:osd.3 smithi146 running (2m) 66s ago 2m 41.5M 4970M 18.2.1 5be31c24972a 1739155eb370 2024-05-04T12:14:48.036 INFO:teuthology.orchestra.run.smithi079.stdout:osd.4 smithi146 running (99s) 66s ago 98s 40.9M 4970M 18.2.1 5be31c24972a be73d59d6afe 2024-05-04T12:14:48.036 INFO:teuthology.orchestra.run.smithi079.stdout:osd.5 smithi146 running (72s) 66s ago 71s 13.8M 4970M 18.2.1 5be31c24972a ce30a9236f6c 2024-05-04T12:14:48.036 INFO:teuthology.orchestra.run.smithi079.stdout:prometheus.smithi079 smithi079 *:9095 running (3m) 2m ago 4m 40.2M - 2.43.0 a07b618ecd1d 1b5e4dd69692 2024-05-04T12:14:48.036 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:48.035+0000 7f24802da700 1 -- 172.21.15.79:0/3720424769 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f246006c4e0 msgr2=0x7f246006e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:14:48.037 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:48.035+0000 7f24802da700 1 --2- 172.21.15.79:0/3720424769 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f246006c4e0 0x7f246006e9a0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f2478101d10 tx=0x7f247000a380 comp rx=0 tx=0).stop 2024-05-04T12:14:48.037 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:48.035+0000 7f24802da700 1 -- 172.21.15.79:0/3720424769 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2478101eb0 msgr2=0x7f24781969f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:14:48.037 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:48.035+0000 7f24802da700 1 --2- 172.21.15.79:0/3720424769 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2478101eb0 0x7f24781969f0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f246c006010 tx=0x7f246c00bd70 comp rx=0 tx=0).stop 2024-05-04T12:14:48.037 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:48.035+0000 7f24802da700 1 -- 172.21.15.79:0/3720424769 shutdown_connections 2024-05-04T12:14:48.037 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:48.035+0000 7f24802da700 1 --2- 172.21.15.79:0/3720424769 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f246006c4e0 0x7f246006e9a0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:48.037 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:48.035+0000 7f24802da700 1 --2- 172.21.15.79:0/3720424769 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2478100cb0 0x7f24781964b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:48.037 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:48.035+0000 7f24802da700 1 --2- 172.21.15.79:0/3720424769 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2478101eb0 0x7f24781969f0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:48.037 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:48.035+0000 7f24802da700 1 -- 172.21.15.79:0/3720424769 >> 172.21.15.79:0/3720424769 conn(0x7f24780fc230 msgr2=0x7f24781050e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:14:48.037 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:48.035+0000 7f24802da700 1 -- 172.21.15.79:0/3720424769 shutdown_connections 2024-05-04T12:14:48.037 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:48.035+0000 7f24802da700 1 -- 172.21.15.79:0/3720424769 wait complete. 2024-05-04T12:14:48.687 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d9b985c4-0a0e-11ef-bc97-c7b262605968 -- bash -c 'ceph orch ls' 2024-05-04T12:14:48.894 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/d9b985c4-0a0e-11ef-bc97-c7b262605968/mon.smithi079/config 2024-05-04T12:14:49.324 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:14:49 smithi079 ceph-mon[27222]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-04T12:14:49.324 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:14:49 smithi079 ceph-mon[27222]: from='client.14446 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-04T12:14:49.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:14:49 smithi146 ceph-mon[35311]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-04T12:14:49.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:14:49 smithi146 ceph-mon[35311]: from='client.14446 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-04T12:14:50.406 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:50.407+0000 7eff951a2700 1 -- 172.21.15.79:0/4286644556 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7eff90100e10 msgr2=0x7eff90101270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:14:50.406 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:50.407+0000 7eff951a2700 1 --2- 172.21.15.79:0/4286644556 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7eff90100e10 0x7eff90101270 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7eff84009870 tx=0x7eff84009b80 comp rx=0 tx=0).stop 2024-05-04T12:14:50.406 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:50.407+0000 7eff951a2700 1 -- 172.21.15.79:0/4286644556 shutdown_connections 2024-05-04T12:14:50.406 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:50.407+0000 7eff951a2700 1 --2- 172.21.15.79:0/4286644556 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7eff90100e10 0x7eff90101270 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:50.406 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:50.407+0000 7eff951a2700 1 --2- 172.21.15.79:0/4286644556 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7eff900ffc10 0x7eff90100030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:50.406 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:50.407+0000 7eff951a2700 1 -- 172.21.15.79:0/4286644556 >> 172.21.15.79:0/4286644556 conn(0x7eff900fb190 msgr2=0x7eff900fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:14:50.406 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:50.407+0000 7eff951a2700 1 -- 172.21.15.79:0/4286644556 shutdown_connections 2024-05-04T12:14:50.406 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:50.407+0000 7eff951a2700 1 -- 172.21.15.79:0/4286644556 wait complete. 2024-05-04T12:14:50.406 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:50.408+0000 7eff951a2700 1 Processor -- start 2024-05-04T12:14:50.408 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:50.408+0000 7eff951a2700 1 -- start start 2024-05-04T12:14:50.408 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:50.408+0000 7eff951a2700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7eff900ffc10 0x7eff900690e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:14:50.408 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:50.408+0000 7eff951a2700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7eff90100e10 0x7eff9019b200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:14:50.408 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:50.408+0000 7eff951a2700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff9019b7d0 con 0x7eff900ffc10 2024-05-04T12:14:50.408 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:50.408+0000 7eff951a2700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff9019b910 con 0x7eff90100e10 2024-05-04T12:14:50.408 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:50.408+0000 7eff8ed9d700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7eff900ffc10 0x7eff900690e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:14:50.408 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:50.408+0000 7eff8ed9d700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7eff900ffc10 0x7eff900690e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:33018/0 (socket says 172.21.15.79:33018) 2024-05-04T12:14:50.408 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:50.408+0000 7eff8ed9d700 1 -- 172.21.15.79:0/1160419298 learned_addr learned my addr 172.21.15.79:0/1160419298 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:14:50.408 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:50.408+0000 7eff8e59c700 1 --2- 172.21.15.79:0/1160419298 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7eff90100e10 0x7eff9019b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:14:50.408 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:50.409+0000 7eff8ed9d700 1 -- 172.21.15.79:0/1160419298 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7eff90100e10 msgr2=0x7eff9019b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:14:50.408 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:50.409+0000 7eff8ed9d700 1 --2- 172.21.15.79:0/1160419298 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7eff90100e10 0x7eff9019b200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:50.408 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:50.409+0000 7eff8ed9d700 1 -- 172.21.15.79:0/1160419298 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff84009520 con 0x7eff900ffc10 2024-05-04T12:14:50.408 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:50.409+0000 7eff8ed9d700 1 --2- 172.21.15.79:0/1160419298 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7eff900ffc10 0x7eff900690e0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7eff8800e970 tx=0x7eff8800ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:14:50.408 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:50.409+0000 7eff77fff700 1 -- 172.21.15.79:0/1160419298 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7eff880041f0 con 0x7eff900ffc10 2024-05-04T12:14:50.409 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:50.409+0000 7eff951a2700 1 -- 172.21.15.79:0/1160419298 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff9019bbf0 con 0x7eff900ffc10 2024-05-04T12:14:50.409 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:50.409+0000 7eff77fff700 1 -- 172.21.15.79:0/1160419298 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7eff8800cc70 con 0x7eff900ffc10 2024-05-04T12:14:50.409 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:50.409+0000 7eff77fff700 1 -- 172.21.15.79:0/1160419298 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7eff88003c90 con 0x7eff900ffc10 2024-05-04T12:14:50.409 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:50.409+0000 7eff951a2700 1 -- 172.21.15.79:0/1160419298 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff9019c110 con 0x7eff900ffc10 2024-05-04T12:14:50.409 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:50.410+0000 7eff951a2700 1 -- 172.21.15.79:0/1160419298 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff90108700 con 0x7eff900ffc10 2024-05-04T12:14:50.409 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:50.410+0000 7eff77fff700 1 -- 172.21.15.79:0/1160419298 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7eff88018430 con 0x7eff900ffc10 2024-05-04T12:14:50.410 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:50.411+0000 7eff77fff700 1 --2- 172.21.15.79:0/1160419298 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7eff7806c4e0 0x7eff7806e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:14:50.410 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:50.411+0000 7eff77fff700 1 -- 172.21.15.79:0/1160419298 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7eff88014070 con 0x7eff900ffc10 2024-05-04T12:14:50.410 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:50.411+0000 7eff8e59c700 1 --2- 172.21.15.79:0/1160419298 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7eff7806c4e0 0x7eff7806e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:14:50.410 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:50.411+0000 7eff8e59c700 1 --2- 172.21.15.79:0/1160419298 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7eff7806c4e0 0x7eff7806e9a0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7eff8400b6f0 tx=0x7eff84019040 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:14:50.413 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:50.415+0000 7eff77fff700 1 -- 172.21.15.79:0/1160419298 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7eff88059d20 con 0x7eff900ffc10 2024-05-04T12:14:50.609 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:50.611+0000 7eff951a2700 1 -- 172.21.15.79:0/1160419298 --> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) v1 -- 0x7eff90105760 con 0x7eff7806c4e0 2024-05-04T12:14:50.614 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:50.616+0000 7eff77fff700 1 -- 172.21.15.79:0/1160419298 <== mgr.14219 v2:172.21.15.79:6800/402764217 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+1300 (secure 0 0 0) 0x7eff90105760 con 0x7eff7806c4e0 2024-05-04T12:14:50.614 INFO:teuthology.orchestra.run.smithi079.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-05-04T12:14:50.614 INFO:teuthology.orchestra.run.smithi079.stdout:alertmanager ?:9093,9094 1/1 2m ago 6m count:1 2024-05-04T12:14:50.614 INFO:teuthology.orchestra.run.smithi079.stdout:ceph-exporter 2/2 2m ago 6m * 2024-05-04T12:14:50.614 INFO:teuthology.orchestra.run.smithi079.stdout:crash 2/2 2m ago 6m * 2024-05-04T12:14:50.614 INFO:teuthology.orchestra.run.smithi079.stdout:grafana ?:3000 1/1 2m ago 6m count:1 2024-05-04T12:14:50.614 INFO:teuthology.orchestra.run.smithi079.stdout:mgr 2/2 2m ago 6m count:2 2024-05-04T12:14:50.614 INFO:teuthology.orchestra.run.smithi079.stdout:mon 2/2 2m ago 5m smithi079:172.21.15.79=smithi079;smithi146:172.21.15.146=smithi146;count:2 2024-05-04T12:14:50.614 INFO:teuthology.orchestra.run.smithi079.stdout:node-exporter ?:9100 2/2 2m ago 6m * 2024-05-04T12:14:50.614 INFO:teuthology.orchestra.run.smithi079.stdout:osd 6 2m ago - 2024-05-04T12:14:50.615 INFO:teuthology.orchestra.run.smithi079.stdout:prometheus ?:9095 1/1 2m ago 6m count:1 2024-05-04T12:14:50.616 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:50.618+0000 7eff951a2700 1 -- 172.21.15.79:0/1160419298 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7eff7806c4e0 msgr2=0x7eff7806e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:14:50.616 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:50.618+0000 7eff951a2700 1 --2- 172.21.15.79:0/1160419298 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7eff7806c4e0 0x7eff7806e9a0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7eff8400b6f0 tx=0x7eff84019040 comp rx=0 tx=0).stop 2024-05-04T12:14:50.616 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:50.618+0000 7eff951a2700 1 -- 172.21.15.79:0/1160419298 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7eff900ffc10 msgr2=0x7eff900690e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:14:50.616 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:50.618+0000 7eff951a2700 1 --2- 172.21.15.79:0/1160419298 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7eff900ffc10 0x7eff900690e0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7eff8800e970 tx=0x7eff8800ec80 comp rx=0 tx=0).stop 2024-05-04T12:14:50.616 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:50.618+0000 7eff951a2700 1 -- 172.21.15.79:0/1160419298 shutdown_connections 2024-05-04T12:14:50.616 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:50.618+0000 7eff951a2700 1 --2- 172.21.15.79:0/1160419298 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7eff7806c4e0 0x7eff7806e9a0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:50.616 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:50.618+0000 7eff951a2700 1 --2- 172.21.15.79:0/1160419298 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7eff900ffc10 0x7eff900690e0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:50.616 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:50.618+0000 7eff951a2700 1 --2- 172.21.15.79:0/1160419298 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7eff90100e10 0x7eff9019b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:50.616 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:50.618+0000 7eff951a2700 1 -- 172.21.15.79:0/1160419298 >> 172.21.15.79:0/1160419298 conn(0x7eff900fb190 msgr2=0x7eff90104040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:14:50.616 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:50.618+0000 7eff951a2700 1 -- 172.21.15.79:0/1160419298 shutdown_connections 2024-05-04T12:14:50.616 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:50.618+0000 7eff951a2700 1 -- 172.21.15.79:0/1160419298 wait complete. 2024-05-04T12:14:51.192 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:14:51 smithi079 ceph-mon[27222]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-04T12:14:51.214 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d9b985c4-0a0e-11ef-bc97-c7b262605968 -- bash -c 'ceph orch host ls' 2024-05-04T12:14:51.418 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/d9b985c4-0a0e-11ef-bc97-c7b262605968/mon.smithi079/config 2024-05-04T12:14:51.526 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:14:51 smithi146 ceph-mon[35311]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-04T12:14:52.195 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:14:52 smithi079 ceph-mon[27222]: from='client.14450 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-04T12:14:52.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:14:52 smithi146 ceph-mon[35311]: from='client.14450 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-04T12:14:53.015 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:53.014+0000 7f27eff00700 1 -- 172.21.15.79:0/3402493503 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f27e8101eb0 msgr2=0x7f27e8104340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:14:53.015 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:53.014+0000 7f27eff00700 1 --2- 172.21.15.79:0/3402493503 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f27e8101eb0 0x7f27e8104340 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f27dc009910 tx=0x7f27dc009c20 comp rx=0 tx=0).stop 2024-05-04T12:14:53.015 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:53.014+0000 7f27eff00700 1 -- 172.21.15.79:0/3402493503 shutdown_connections 2024-05-04T12:14:53.015 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:53.014+0000 7f27eff00700 1 --2- 172.21.15.79:0/3402493503 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f27e8101eb0 0x7f27e8104340 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:53.015 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:53.014+0000 7f27eff00700 1 --2- 172.21.15.79:0/3402493503 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f27e80ff540 0x7f27e8101970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:53.016 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:53.014+0000 7f27eff00700 1 -- 172.21.15.79:0/3402493503 >> 172.21.15.79:0/3402493503 conn(0x7f27e80fb160 msgr2=0x7f27e80fd5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:14:53.016 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:53.014+0000 7f27eff00700 1 -- 172.21.15.79:0/3402493503 shutdown_connections 2024-05-04T12:14:53.016 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:53.014+0000 7f27eff00700 1 -- 172.21.15.79:0/3402493503 wait complete. 2024-05-04T12:14:53.016 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:53.014+0000 7f27eff00700 1 Processor -- start 2024-05-04T12:14:53.016 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:53.014+0000 7f27eff00700 1 -- start start 2024-05-04T12:14:53.016 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:53.014+0000 7f27eff00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f27e80ff540 0x7f27e819cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:14:53.016 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:53.014+0000 7f27eff00700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f27e8101eb0 0x7f27e819d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:14:53.016 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:53.014+0000 7f27eff00700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27e819d900 con 0x7f27e8101eb0 2024-05-04T12:14:53.016 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:53.014+0000 7f27eff00700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27e819da40 con 0x7f27e80ff540 2024-05-04T12:14:53.016 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:53.015+0000 7f27ed49b700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f27e8101eb0 0x7f27e819d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:14:53.017 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:53.015+0000 7f27ed49b700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f27e8101eb0 0x7f27e819d2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:33030/0 (socket says 172.21.15.79:33030) 2024-05-04T12:14:53.017 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:53.015+0000 7f27ed49b700 1 -- 172.21.15.79:0/1384183287 learned_addr learned my addr 172.21.15.79:0/1384183287 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:14:53.017 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:53.015+0000 7f27edc9c700 1 --2- 172.21.15.79:0/1384183287 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f27e80ff540 0x7f27e819cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:14:53.017 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:53.015+0000 7f27ed49b700 1 -- 172.21.15.79:0/1384183287 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f27e80ff540 msgr2=0x7f27e819cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:14:53.017 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:53.015+0000 7f27ed49b700 1 --2- 172.21.15.79:0/1384183287 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f27e80ff540 0x7f27e819cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:53.017 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:53.015+0000 7f27ed49b700 1 -- 172.21.15.79:0/1384183287 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27dc0095f0 con 0x7f27e8101eb0 2024-05-04T12:14:53.017 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:53.015+0000 7f27ed49b700 1 --2- 172.21.15.79:0/1384183287 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f27e8101eb0 0x7f27e819d2e0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f27dc006010 tx=0x7f27dc00bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:14:53.017 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:53.015+0000 7f27e6ffd700 1 -- 172.21.15.79:0/1384183287 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f27dc01d070 con 0x7f27e8101eb0 2024-05-04T12:14:53.017 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:53.015+0000 7f27eff00700 1 -- 172.21.15.79:0/1384183287 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27e81a2490 con 0x7f27e8101eb0 2024-05-04T12:14:53.017 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:53.015+0000 7f27e6ffd700 1 -- 172.21.15.79:0/1384183287 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f27dc00f460 con 0x7f27e8101eb0 2024-05-04T12:14:53.017 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:53.015+0000 7f27e6ffd700 1 -- 172.21.15.79:0/1384183287 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f27dc021640 con 0x7f27e8101eb0 2024-05-04T12:14:53.017 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:53.015+0000 7f27eff00700 1 -- 172.21.15.79:0/1384183287 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27e81a2980 con 0x7f27e8101eb0 2024-05-04T12:14:53.018 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:53.016+0000 7f27eff00700 1 -- 172.21.15.79:0/1384183287 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27e8196c20 con 0x7f27e8101eb0 2024-05-04T12:14:53.018 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:53.017+0000 7f27e6ffd700 1 -- 172.21.15.79:0/1384183287 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f27dc00fab0 con 0x7f27e8101eb0 2024-05-04T12:14:53.018 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:53.017+0000 7f27e6ffd700 1 --2- 172.21.15.79:0/1384183287 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f27d006c4e0 0x7f27d006e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:14:53.018 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:53.017+0000 7f27e6ffd700 1 -- 172.21.15.79:0/1384183287 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f27dc08c730 con 0x7f27e8101eb0 2024-05-04T12:14:53.019 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:53.017+0000 7f27edc9c700 1 --2- 172.21.15.79:0/1384183287 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f27d006c4e0 0x7f27d006e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:14:53.019 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:53.018+0000 7f27edc9c700 1 --2- 172.21.15.79:0/1384183287 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f27d006c4e0 0x7f27d006e9a0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f27e0005950 tx=0x7f27e000b410 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:14:53.021 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:53.022+0000 7f27e6ffd700 1 -- 172.21.15.79:0/1384183287 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f27dc05ace0 con 0x7f27e8101eb0 2024-05-04T12:14:53.219 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:53.220+0000 7f27eff00700 1 -- 172.21.15.79:0/1384183287 --> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f27e802d050 con 0x7f27d006c4e0 2024-05-04T12:14:53.220 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:53.221+0000 7f27e6ffd700 1 -- 172.21.15.79:0/1384183287 <== mgr.14219 v2:172.21.15.79:6800/402764217 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+148 (secure 0 0 0) 0x7f27e802d050 con 0x7f27d006c4e0 2024-05-04T12:14:53.221 INFO:teuthology.orchestra.run.smithi079.stdout:HOST ADDR LABELS STATUS 2024-05-04T12:14:53.221 INFO:teuthology.orchestra.run.smithi079.stdout:smithi079 172.21.15.79 2024-05-04T12:14:53.221 INFO:teuthology.orchestra.run.smithi079.stdout:smithi146 172.21.15.146 2024-05-04T12:14:53.221 INFO:teuthology.orchestra.run.smithi079.stdout:2 hosts in cluster 2024-05-04T12:14:53.223 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:53.223+0000 7f27eff00700 1 -- 172.21.15.79:0/1384183287 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f27d006c4e0 msgr2=0x7f27d006e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:14:53.223 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:53.223+0000 7f27eff00700 1 --2- 172.21.15.79:0/1384183287 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f27d006c4e0 0x7f27d006e9a0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f27e0005950 tx=0x7f27e000b410 comp rx=0 tx=0).stop 2024-05-04T12:14:53.223 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:53.223+0000 7f27eff00700 1 -- 172.21.15.79:0/1384183287 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f27e8101eb0 msgr2=0x7f27e819d2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:14:53.223 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:53.223+0000 7f27eff00700 1 --2- 172.21.15.79:0/1384183287 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f27e8101eb0 0x7f27e819d2e0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f27dc006010 tx=0x7f27dc00bd70 comp rx=0 tx=0).stop 2024-05-04T12:14:53.223 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:53.223+0000 7f27eff00700 1 -- 172.21.15.79:0/1384183287 shutdown_connections 2024-05-04T12:14:53.223 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:53.223+0000 7f27eff00700 1 --2- 172.21.15.79:0/1384183287 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f27d006c4e0 0x7f27d006e9a0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:53.223 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:53.223+0000 7f27eff00700 1 --2- 172.21.15.79:0/1384183287 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f27e80ff540 0x7f27e819cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:53.223 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:53.223+0000 7f27eff00700 1 --2- 172.21.15.79:0/1384183287 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f27e8101eb0 0x7f27e819d2e0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:53.223 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:53.223+0000 7f27eff00700 1 -- 172.21.15.79:0/1384183287 >> 172.21.15.79:0/1384183287 conn(0x7f27e80fb160 msgr2=0x7f27e80fd5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:14:53.223 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:53.224+0000 7f27eff00700 1 -- 172.21.15.79:0/1384183287 shutdown_connections 2024-05-04T12:14:53.223 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:53.224+0000 7f27eff00700 1 -- 172.21.15.79:0/1384183287 wait complete. 2024-05-04T12:14:53.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:14:53 smithi079 ceph-mon[27222]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-04T12:14:53.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:14:53 smithi146 ceph-mon[35311]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-04T12:14:53.870 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d9b985c4-0a0e-11ef-bc97-c7b262605968 -- bash -c 'ceph orch device ls' 2024-05-04T12:14:54.076 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/d9b985c4-0a0e-11ef-bc97-c7b262605968/mon.smithi079/config 2024-05-04T12:14:55.195 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:14:55 smithi079 ceph-mon[27222]: from='client.14454 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-04T12:14:55.195 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:14:55 smithi079 ceph-mon[27222]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-04T12:14:55.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:14:55 smithi146 ceph-mon[35311]: from='client.14454 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-04T12:14:55.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:14:55 smithi146 ceph-mon[35311]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-04T12:14:55.630 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:55.629+0000 7f951c6cc700 1 -- 172.21.15.79:0/2042928894 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9514100cb0 msgr2=0x7f95141010d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:14:55.630 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:55.629+0000 7f951c6cc700 1 --2- 172.21.15.79:0/2042928894 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9514100cb0 0x7f95141010d0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f9510009910 tx=0x7f9510009c20 comp rx=0 tx=0).stop 2024-05-04T12:14:55.630 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:55.630+0000 7f951c6cc700 1 -- 172.21.15.79:0/2042928894 shutdown_connections 2024-05-04T12:14:55.631 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:55.630+0000 7f951c6cc700 1 --2- 172.21.15.79:0/2042928894 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9514101eb0 0x7f9514102310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:55.631 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:55.630+0000 7f951c6cc700 1 --2- 172.21.15.79:0/2042928894 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9514100cb0 0x7f95141010d0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:55.631 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:55.630+0000 7f951c6cc700 1 -- 172.21.15.79:0/2042928894 >> 172.21.15.79:0/2042928894 conn(0x7f95140fc230 msgr2=0x7f95140fe690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:14:55.631 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:55.630+0000 7f951c6cc700 1 -- 172.21.15.79:0/2042928894 shutdown_connections 2024-05-04T12:14:55.631 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:55.630+0000 7f951c6cc700 1 -- 172.21.15.79:0/2042928894 wait complete. 2024-05-04T12:14:55.631 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:55.630+0000 7f951c6cc700 1 Processor -- start 2024-05-04T12:14:55.631 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:55.630+0000 7f951c6cc700 1 -- start start 2024-05-04T12:14:55.631 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:55.630+0000 7f951c6cc700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9514100cb0 0x7f95141964c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:14:55.631 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:55.630+0000 7f951c6cc700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9514101eb0 0x7f9514196a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:14:55.631 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:55.630+0000 7f951c6cc700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9514196f90 con 0x7f9514100cb0 2024-05-04T12:14:55.631 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:55.630+0000 7f951c6cc700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95141970d0 con 0x7f9514101eb0 2024-05-04T12:14:55.632 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:55.630+0000 7f951a468700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9514100cb0 0x7f95141964c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:14:55.632 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:55.630+0000 7f951a468700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9514100cb0 0x7f95141964c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:52376/0 (socket says 172.21.15.79:52376) 2024-05-04T12:14:55.632 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:55.630+0000 7f951a468700 1 -- 172.21.15.79:0/4044088876 learned_addr learned my addr 172.21.15.79:0/4044088876 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:14:55.632 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:55.631+0000 7f951a468700 1 -- 172.21.15.79:0/4044088876 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9514101eb0 msgr2=0x7f9514196a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:14:55.632 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:55.631+0000 7f951a468700 1 --2- 172.21.15.79:0/4044088876 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9514101eb0 0x7f9514196a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:55.632 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:55.631+0000 7f951a468700 1 -- 172.21.15.79:0/4044088876 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95100095f0 con 0x7f9514100cb0 2024-05-04T12:14:55.632 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:55.631+0000 7f951a468700 1 --2- 172.21.15.79:0/4044088876 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9514100cb0 0x7f95141964c0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f951000bda0 tx=0x7f951000be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:14:55.632 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:55.631+0000 7f950b7fe700 1 -- 172.21.15.79:0/4044088876 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f951001d070 con 0x7f9514100cb0 2024-05-04T12:14:55.632 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:55.631+0000 7f950b7fe700 1 -- 172.21.15.79:0/4044088876 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f9510004c50 con 0x7f9514100cb0 2024-05-04T12:14:55.632 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:55.631+0000 7f951c6cc700 1 -- 172.21.15.79:0/4044088876 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f951419bb30 con 0x7f9514100cb0 2024-05-04T12:14:55.632 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:55.631+0000 7f950b7fe700 1 -- 172.21.15.79:0/4044088876 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f951000f690 con 0x7f9514100cb0 2024-05-04T12:14:55.632 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:55.631+0000 7f951c6cc700 1 -- 172.21.15.79:0/4044088876 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f951419bff0 con 0x7f9514100cb0 2024-05-04T12:14:55.633 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:55.632+0000 7f951c6cc700 1 -- 172.21.15.79:0/4044088876 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9514109810 con 0x7f9514100cb0 2024-05-04T12:14:55.633 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:55.633+0000 7f950b7fe700 1 -- 172.21.15.79:0/4044088876 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f9510022a20 con 0x7f9514100cb0 2024-05-04T12:14:55.633 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:55.633+0000 7f950b7fe700 1 --2- 172.21.15.79:0/4044088876 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f950006c4e0 0x7f950006e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:14:55.633 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:55.633+0000 7f9519c67700 1 --2- 172.21.15.79:0/4044088876 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f950006c4e0 0x7f950006e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:14:55.633 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:55.633+0000 7f950b7fe700 1 -- 172.21.15.79:0/4044088876 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f951008caa0 con 0x7f9514100cb0 2024-05-04T12:14:55.633 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:55.633+0000 7f9519c67700 1 --2- 172.21.15.79:0/4044088876 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f950006c4e0 0x7f950006e9a0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f9504005fd0 tx=0x7f9504005d00 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:14:55.636 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:55.637+0000 7f950b7fe700 1 -- 172.21.15.79:0/4044088876 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f951005b2a0 con 0x7f9514100cb0 2024-05-04T12:14:55.834 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:55.835+0000 7f951c6cc700 1 -- 172.21.15.79:0/4044088876 --> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f9514106800 con 0x7f950006c4e0 2024-05-04T12:14:55.835 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:55.836+0000 7f950b7fe700 1 -- 172.21.15.79:0/4044088876 <== mgr.14219 v2:172.21.15.79:6800/402764217 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+399 (secure 0 0 0) 0x7f9514106800 con 0x7f950006c4e0 2024-05-04T12:14:55.835 INFO:teuthology.orchestra.run.smithi079.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-05-04T12:14:55.836 INFO:teuthology.orchestra.run.smithi079.stdout:smithi079 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT6204013G400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-05-04T12:14:55.836 INFO:teuthology.orchestra.run.smithi079.stdout:smithi146 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT623300GB400BGN 372G No 71s ago Has a FileSystem, LVM detected 2024-05-04T12:14:55.838 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:55.838+0000 7f951c6cc700 1 -- 172.21.15.79:0/4044088876 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f950006c4e0 msgr2=0x7f950006e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:14:55.838 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:55.838+0000 7f951c6cc700 1 --2- 172.21.15.79:0/4044088876 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f950006c4e0 0x7f950006e9a0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f9504005fd0 tx=0x7f9504005d00 comp rx=0 tx=0).stop 2024-05-04T12:14:55.838 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:55.838+0000 7f951c6cc700 1 -- 172.21.15.79:0/4044088876 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9514100cb0 msgr2=0x7f95141964c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:14:55.838 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:55.838+0000 7f951c6cc700 1 --2- 172.21.15.79:0/4044088876 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9514100cb0 0x7f95141964c0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f951000bda0 tx=0x7f951000be80 comp rx=0 tx=0).stop 2024-05-04T12:14:55.838 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:55.838+0000 7f951c6cc700 1 -- 172.21.15.79:0/4044088876 shutdown_connections 2024-05-04T12:14:55.838 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:55.838+0000 7f951c6cc700 1 --2- 172.21.15.79:0/4044088876 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f950006c4e0 0x7f950006e9a0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:55.838 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:55.838+0000 7f951c6cc700 1 --2- 172.21.15.79:0/4044088876 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9514100cb0 0x7f95141964c0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:55.839 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:55.838+0000 7f951c6cc700 1 --2- 172.21.15.79:0/4044088876 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9514101eb0 0x7f9514196a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:55.839 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:55.838+0000 7f951c6cc700 1 -- 172.21.15.79:0/4044088876 >> 172.21.15.79:0/4044088876 conn(0x7f95140fc230 msgr2=0x7f95141050e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:14:55.839 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:55.838+0000 7f951c6cc700 1 -- 172.21.15.79:0/4044088876 shutdown_connections 2024-05-04T12:14:55.839 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:55.838+0000 7f951c6cc700 1 -- 172.21.15.79:0/4044088876 wait complete. 2024-05-04T12:14:56.458 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-05-04T12:14:56.468 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi079.front.sepia.ceph.com 2024-05-04T12:14:56.469 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d9b985c4-0a0e-11ef-bc97-c7b262605968 -- bash -c 'ceph fs volume create cephfs --placement=4' 2024-05-04T12:14:56.673 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/d9b985c4-0a0e-11ef-bc97-c7b262605968/mon.smithi079/config 2024-05-04T12:14:57.390 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:14:57 smithi079 ceph-mon[27222]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-04T12:14:57.390 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:14:57 smithi079 ceph-mon[27222]: from='client.14458 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-04T12:14:57.391 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:14:57 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-04T12:14:57.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:14:57 smithi146 ceph-mon[35311]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-04T12:14:57.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:14:57 smithi146 ceph-mon[35311]: from='client.14458 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-04T12:14:57.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:14:57 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-04T12:14:58.198 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:58.175+0000 7f15b8d4f700 1 -- 172.21.15.79:0/3488969413 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f15b4100dc0 msgr2=0x7f15b4101220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:14:58.198 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:58.175+0000 7f15b8d4f700 1 --2- 172.21.15.79:0/3488969413 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f15b4100dc0 0x7f15b4101220 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f15a4009910 tx=0x7f15a4009c20 comp rx=0 tx=0).stop 2024-05-04T12:14:58.198 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:58.175+0000 7f15b8d4f700 1 -- 172.21.15.79:0/3488969413 shutdown_connections 2024-05-04T12:14:58.198 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:58.175+0000 7f15b8d4f700 1 --2- 172.21.15.79:0/3488969413 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f15b4100dc0 0x7f15b4101220 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:58.198 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:58.175+0000 7f15b8d4f700 1 --2- 172.21.15.79:0/3488969413 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f15b40ffbc0 0x7f15b40fffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:58.199 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:58.175+0000 7f15b8d4f700 1 -- 172.21.15.79:0/3488969413 >> 172.21.15.79:0/3488969413 conn(0x7f15b40fb160 msgr2=0x7f15b40fd5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:14:58.199 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:58.175+0000 7f15b8d4f700 1 -- 172.21.15.79:0/3488969413 shutdown_connections 2024-05-04T12:14:58.199 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:58.175+0000 7f15b8d4f700 1 -- 172.21.15.79:0/3488969413 wait complete. 2024-05-04T12:14:58.199 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:58.176+0000 7f15b8d4f700 1 Processor -- start 2024-05-04T12:14:58.199 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:58.176+0000 7f15b8d4f700 1 -- start start 2024-05-04T12:14:58.199 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:58.176+0000 7f15b8d4f700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f15b40ffbc0 0x7f15b41956c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:14:58.199 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:58.176+0000 7f15b8d4f700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f15b4100dc0 0x7f15b4195c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:14:58.199 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:58.176+0000 7f15b8d4f700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15b4196190 con 0x7f15b4100dc0 2024-05-04T12:14:58.199 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:58.176+0000 7f15b8d4f700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15b41962d0 con 0x7f15b40ffbc0 2024-05-04T12:14:58.200 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:58.176+0000 7f15b1d9b700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f15b4100dc0 0x7f15b4195c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:14:58.200 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:58.176+0000 7f15b1d9b700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f15b4100dc0 0x7f15b4195c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:52386/0 (socket says 172.21.15.79:52386) 2024-05-04T12:14:58.200 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:58.176+0000 7f15b1d9b700 1 -- 172.21.15.79:0/1894999123 learned_addr learned my addr 172.21.15.79:0/1894999123 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:14:58.200 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:58.177+0000 7f15b259c700 1 --2- 172.21.15.79:0/1894999123 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f15b40ffbc0 0x7f15b41956c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:14:58.200 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:58.177+0000 7f15b1d9b700 1 -- 172.21.15.79:0/1894999123 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f15b40ffbc0 msgr2=0x7f15b41956c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:14:58.200 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:58.177+0000 7f15b1d9b700 1 --2- 172.21.15.79:0/1894999123 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f15b40ffbc0 0x7f15b41956c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:14:58.200 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:58.177+0000 7f15b1d9b700 1 -- 172.21.15.79:0/1894999123 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15a40095f0 con 0x7f15b4100dc0 2024-05-04T12:14:58.200 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:58.177+0000 7f15b1d9b700 1 --2- 172.21.15.79:0/1894999123 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f15b4100dc0 0x7f15b4195c00 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f15a400bd70 tx=0x7f15a400be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:14:58.200 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:58.177+0000 7f15ab7fe700 1 -- 172.21.15.79:0/1894999123 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f15a401d070 con 0x7f15b4100dc0 2024-05-04T12:14:58.200 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:58.177+0000 7f15ab7fe700 1 -- 172.21.15.79:0/1894999123 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f15a4022470 con 0x7f15b4100dc0 2024-05-04T12:14:58.200 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:58.177+0000 7f15b8d4f700 1 -- 172.21.15.79:0/1894999123 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15b419ad30 con 0x7f15b4100dc0 2024-05-04T12:14:58.200 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:58.177+0000 7f15ab7fe700 1 -- 172.21.15.79:0/1894999123 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f15a400f670 con 0x7f15b4100dc0 2024-05-04T12:14:58.200 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:58.177+0000 7f15b8d4f700 1 -- 172.21.15.79:0/1894999123 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15b419b1f0 con 0x7f15b4100dc0 2024-05-04T12:14:58.200 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:58.178+0000 7f15b8d4f700 1 -- 172.21.15.79:0/1894999123 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15b418f530 con 0x7f15b4100dc0 2024-05-04T12:14:58.201 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:58.179+0000 7f15ab7fe700 1 -- 172.21.15.79:0/1894999123 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f15a4022a50 con 0x7f15b4100dc0 2024-05-04T12:14:58.201 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:58.179+0000 7f15ab7fe700 1 --2- 172.21.15.79:0/1894999123 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f15a006c4e0 0x7f15a006e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:14:58.201 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:58.179+0000 7f15ab7fe700 1 -- 172.21.15.79:0/1894999123 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f15a408cb50 con 0x7f15b4100dc0 2024-05-04T12:14:58.201 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:58.179+0000 7f15b259c700 1 --2- 172.21.15.79:0/1894999123 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f15a006c4e0 0x7f15a006e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:14:58.201 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:58.179+0000 7f15b259c700 1 --2- 172.21.15.79:0/1894999123 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f15a006c4e0 0x7f15a006e9a0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f159c005fd0 tx=0x7f159c005e50 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:14:58.201 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:58.183+0000 7f15ab7fe700 1 -- 172.21.15.79:0/1894999123 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f15a405b1b0 con 0x7f15b4100dc0 2024-05-04T12:14:58.400 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:14:58.400+0000 7f15b8d4f700 1 -- 172.21.15.79:0/1894999123 --> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] -- mgr_command(tid 0: {"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7f15b402d050 con 0x7f15a006c4e0 2024-05-04T12:14:59.695 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:14:59 smithi079 ceph-mon[27222]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-04T12:14:59.695 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:14:59 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]: dispatch 2024-05-04T12:14:59.777 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:14:59 smithi146 ceph-mon[35311]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-04T12:14:59.777 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:14:59 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]: dispatch 2024-05-04T12:15:00.106 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:00.107+0000 7f15ab7fe700 1 -- 172.21.15.79:0/1894999123 <== mgr.14219 v2:172.21.15.79:6800/402764217 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f15b402d050 con 0x7f15a006c4e0 2024-05-04T12:15:00.108 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:00.109+0000 7f15b8d4f700 1 -- 172.21.15.79:0/1894999123 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f15a006c4e0 msgr2=0x7f15a006e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:15:00.109 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:00.109+0000 7f15b8d4f700 1 --2- 172.21.15.79:0/1894999123 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f15a006c4e0 0x7f15a006e9a0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f159c005fd0 tx=0x7f159c005e50 comp rx=0 tx=0).stop 2024-05-04T12:15:00.109 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:00.109+0000 7f15b8d4f700 1 -- 172.21.15.79:0/1894999123 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f15b4100dc0 msgr2=0x7f15b4195c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:15:00.109 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:00.109+0000 7f15b8d4f700 1 --2- 172.21.15.79:0/1894999123 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f15b4100dc0 0x7f15b4195c00 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f15a400bd70 tx=0x7f15a400be50 comp rx=0 tx=0).stop 2024-05-04T12:15:00.109 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:00.109+0000 7f15b8d4f700 1 -- 172.21.15.79:0/1894999123 shutdown_connections 2024-05-04T12:15:00.109 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:00.109+0000 7f15b8d4f700 1 --2- 172.21.15.79:0/1894999123 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f15a006c4e0 0x7f15a006e9a0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:00.109 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:00.109+0000 7f15b8d4f700 1 --2- 172.21.15.79:0/1894999123 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f15b40ffbc0 0x7f15b41956c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:00.109 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:00.109+0000 7f15b8d4f700 1 --2- 172.21.15.79:0/1894999123 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f15b4100dc0 0x7f15b4195c00 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:00.109 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:00.109+0000 7f15b8d4f700 1 -- 172.21.15.79:0/1894999123 >> 172.21.15.79:0/1894999123 conn(0x7f15b40fb160 msgr2=0x7f15b4104110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:15:00.109 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:00.109+0000 7f15b8d4f700 1 -- 172.21.15.79:0/1894999123 shutdown_connections 2024-05-04T12:15:00.109 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:00.109+0000 7f15b8d4f700 1 -- 172.21.15.79:0/1894999123 wait complete. 2024-05-04T12:15:00.433 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:15:00.082+0000 7fda5252f700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-04T12:15:00.434 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:00 smithi079 ceph-mon[27222]: from='client.14462 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-04T12:15:00.434 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:00 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]': finished 2024-05-04T12:15:00.434 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:00 smithi079 ceph-mon[27222]: osdmap e41: 6 total, 6 up, 6 in 2024-05-04T12:15:00.434 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:00 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]: dispatch 2024-05-04T12:15:00.434 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:00 smithi079 ceph-mon[27222]: pgmap v173: 33 pgs: 3 creating+peering, 1 active+clean, 29 unknown; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-04T12:15:00.434 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:00 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]': finished 2024-05-04T12:15:00.434 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:00 smithi079 ceph-mon[27222]: osdmap e42: 6 total, 6 up, 6 in 2024-05-04T12:15:00.434 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:00 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]: dispatch 2024-05-04T12:15:00.434 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:00 smithi079 ceph-mon[27222]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-04T12:15:00.434 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:00 smithi079 ceph-mon[27222]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-05-04T12:15:00.434 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:00 smithi079 ceph-mon[27222]: osdmap e43: 6 total, 6 up, 6 in 2024-05-04T12:15:00.434 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:00 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]': finished 2024-05-04T12:15:00.434 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:00 smithi079 ceph-mon[27222]: fsmap cephfs:0 2024-05-04T12:15:00.435 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:00 smithi079 ceph-mon[27222]: Saving service mds.cephfs spec with placement count:4 2024-05-04T12:15:00.435 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:00 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:15:00.435 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:00 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-04T12:15:00.435 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:00 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-04T12:15:00.435 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:00 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-04T12:15:00.435 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:00 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:15:00.435 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:00 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi079.pspich", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-04T12:15:00.435 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:00 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi079.pspich", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-04T12:15:00.435 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:00 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-04T12:15:00.771 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d9b985c4-0a0e-11ef-bc97-c7b262605968 -- bash -c 'ceph fs dump' 2024-05-04T12:15:00.777 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:00 smithi146 ceph-mon[35311]: from='client.14462 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-04T12:15:00.777 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:00 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]': finished 2024-05-04T12:15:00.777 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:00 smithi146 ceph-mon[35311]: osdmap e41: 6 total, 6 up, 6 in 2024-05-04T12:15:00.777 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:00 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]: dispatch 2024-05-04T12:15:00.777 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:00 smithi146 ceph-mon[35311]: pgmap v173: 33 pgs: 3 creating+peering, 1 active+clean, 29 unknown; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-04T12:15:00.777 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:00 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]': finished 2024-05-04T12:15:00.777 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:00 smithi146 ceph-mon[35311]: osdmap e42: 6 total, 6 up, 6 in 2024-05-04T12:15:00.777 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:00 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]: dispatch 2024-05-04T12:15:00.777 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:00 smithi146 ceph-mon[35311]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-04T12:15:00.778 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:00 smithi146 ceph-mon[35311]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-05-04T12:15:00.778 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:00 smithi146 ceph-mon[35311]: osdmap e43: 6 total, 6 up, 6 in 2024-05-04T12:15:00.778 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:00 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]': finished 2024-05-04T12:15:00.778 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:00 smithi146 ceph-mon[35311]: fsmap cephfs:0 2024-05-04T12:15:00.778 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:00 smithi146 ceph-mon[35311]: Saving service mds.cephfs spec with placement count:4 2024-05-04T12:15:00.778 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:00 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:15:00.778 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:00 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-04T12:15:00.778 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:00 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-04T12:15:00.778 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:00 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-04T12:15:00.778 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:00 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:15:00.778 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:00 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi079.pspich", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-04T12:15:00.778 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:00 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi079.pspich", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-04T12:15:00.778 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:00 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-04T12:15:01.690 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:01 smithi079 ceph-mon[27222]: Deploying daemon mds.cephfs.smithi079.pspich on smithi079 2024-05-04T12:15:01.690 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:01 smithi079 ceph-mon[27222]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-04T12:15:01.690 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:01 smithi079 ceph-mon[27222]: osdmap e44: 6 total, 6 up, 6 in 2024-05-04T12:15:01.777 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:01 smithi146 ceph-mon[35311]: Deploying daemon mds.cephfs.smithi079.pspich on smithi079 2024-05-04T12:15:01.777 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:01 smithi146 ceph-mon[35311]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-04T12:15:01.777 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:01 smithi146 ceph-mon[35311]: osdmap e44: 6 total, 6 up, 6 in 2024-05-04T12:15:02.156 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/d9b985c4-0a0e-11ef-bc97-c7b262605968/mon.smithi079/config 2024-05-04T12:15:03.418 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:03 smithi079 ceph-mon[27222]: pgmap v177: 65 pgs: 16 active+clean, 9 creating+peering, 40 unknown; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-04T12:15:03.419 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:03 smithi079 ceph-mon[27222]: osdmap e45: 6 total, 6 up, 6 in 2024-05-04T12:15:03.419 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:03 smithi079 ceph-mon[27222]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-05-04T12:15:03.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:03 smithi146 ceph-mon[35311]: pgmap v177: 65 pgs: 16 active+clean, 9 creating+peering, 40 unknown; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-04T12:15:03.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:03 smithi146 ceph-mon[35311]: osdmap e45: 6 total, 6 up, 6 in 2024-05-04T12:15:03.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:03 smithi146 ceph-mon[35311]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-05-04T12:15:03.964 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:03.963+0000 7fd13d0c5700 1 -- 172.21.15.79:0/2624424568 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd1380ffc10 msgr2=0x7fd138100030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:15:03.964 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:03.963+0000 7fd13d0c5700 1 --2- 172.21.15.79:0/2624424568 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd1380ffc10 0x7fd138100030 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fd12c009910 tx=0x7fd12c009c20 comp rx=0 tx=0).stop 2024-05-04T12:15:03.965 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:03.963+0000 7fd13d0c5700 1 -- 172.21.15.79:0/2624424568 shutdown_connections 2024-05-04T12:15:03.965 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:03.963+0000 7fd13d0c5700 1 --2- 172.21.15.79:0/2624424568 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd138100e10 0x7fd138101270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:03.965 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:03.963+0000 7fd13d0c5700 1 --2- 172.21.15.79:0/2624424568 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd1380ffc10 0x7fd138100030 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:03.965 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:03.963+0000 7fd13d0c5700 1 -- 172.21.15.79:0/2624424568 >> 172.21.15.79:0/2624424568 conn(0x7fd1380fb190 msgr2=0x7fd1380fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:15:03.965 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:03.963+0000 7fd13d0c5700 1 -- 172.21.15.79:0/2624424568 shutdown_connections 2024-05-04T12:15:03.965 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:03.963+0000 7fd13d0c5700 1 -- 172.21.15.79:0/2624424568 wait complete. 2024-05-04T12:15:03.965 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:03.964+0000 7fd13d0c5700 1 Processor -- start 2024-05-04T12:15:03.965 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:03.964+0000 7fd13d0c5700 1 -- start start 2024-05-04T12:15:03.965 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:03.964+0000 7fd13d0c5700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd1380ffc10 0x7fd138196530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:15:03.965 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:03.964+0000 7fd13d0c5700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd138100e10 0x7fd138196a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:15:03.965 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:03.964+0000 7fd13d0c5700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd138197090 con 0x7fd1380ffc10 2024-05-04T12:15:03.965 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:03.964+0000 7fd13d0c5700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1381971d0 con 0x7fd138100e10 2024-05-04T12:15:03.965 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:03.964+0000 7fd136d9d700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd1380ffc10 0x7fd138196530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:15:03.965 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:03.964+0000 7fd136d9d700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd1380ffc10 0x7fd138196530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:56638/0 (socket says 172.21.15.79:56638) 2024-05-04T12:15:03.965 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:03.964+0000 7fd136d9d700 1 -- 172.21.15.79:0/1506945936 learned_addr learned my addr 172.21.15.79:0/1506945936 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:15:03.966 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:03.964+0000 7fd13659c700 1 --2- 172.21.15.79:0/1506945936 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd138100e10 0x7fd138196a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:15:03.966 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:03.965+0000 7fd136d9d700 1 -- 172.21.15.79:0/1506945936 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd138100e10 msgr2=0x7fd138196a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:15:03.966 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:03.965+0000 7fd136d9d700 1 --2- 172.21.15.79:0/1506945936 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd138100e10 0x7fd138196a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:03.966 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:03.965+0000 7fd136d9d700 1 -- 172.21.15.79:0/1506945936 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd12c0095f0 con 0x7fd1380ffc10 2024-05-04T12:15:03.966 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:03.965+0000 7fd136d9d700 1 --2- 172.21.15.79:0/1506945936 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd1380ffc10 0x7fd138196530 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fd12c00bda0 tx=0x7fd12c00be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:15:03.966 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:03.965+0000 7fd11ffff700 1 -- 172.21.15.79:0/1506945936 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd12c01d070 con 0x7fd1380ffc10 2024-05-04T12:15:03.966 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:03.965+0000 7fd11ffff700 1 -- 172.21.15.79:0/1506945936 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fd12c004c50 con 0x7fd1380ffc10 2024-05-04T12:15:03.966 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:03.965+0000 7fd13d0c5700 1 -- 172.21.15.79:0/1506945936 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd13819bc20 con 0x7fd1380ffc10 2024-05-04T12:15:03.966 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:03.965+0000 7fd11ffff700 1 -- 172.21.15.79:0/1506945936 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd12c00f690 con 0x7fd1380ffc10 2024-05-04T12:15:03.966 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:03.965+0000 7fd13d0c5700 1 -- 172.21.15.79:0/1506945936 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd13819c110 con 0x7fd1380ffc10 2024-05-04T12:15:03.967 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:03.966+0000 7fd13d0c5700 1 -- 172.21.15.79:0/1506945936 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd138108780 con 0x7fd1380ffc10 2024-05-04T12:15:03.967 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:03.967+0000 7fd11ffff700 1 -- 172.21.15.79:0/1506945936 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7fd12c00f900 con 0x7fd1380ffc10 2024-05-04T12:15:03.967 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:03.967+0000 7fd11ffff700 1 --2- 172.21.15.79:0/1506945936 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7fd12006c4e0 0x7fd12006e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:15:03.967 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:03.967+0000 7fd13659c700 1 --2- 172.21.15.79:0/1506945936 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7fd12006c4e0 0x7fd12006e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:15:03.967 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:03.967+0000 7fd11ffff700 1 -- 172.21.15.79:0/1506945936 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5468+0+0 (secure 0 0 0) 0x7fd12c08cfe0 con 0x7fd1380ffc10 2024-05-04T12:15:03.967 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:03.967+0000 7fd13659c700 1 --2- 172.21.15.79:0/1506945936 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7fd12006c4e0 0x7fd12006e9a0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fd128006fd0 tx=0x7fd128008040 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:15:03.971 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:03.972+0000 7fd11ffff700 1 -- 172.21.15.79:0/1506945936 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fd12c05b250 con 0x7fd1380ffc10 2024-05-04T12:15:04.205 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:04.205+0000 7fd13d0c5700 1 -- 172.21.15.79:0/1506945936 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fd13819ca70 con 0x7fd1380ffc10 2024-05-04T12:15:04.205 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:04.206+0000 7fd11ffff700 1 -- 172.21.15.79:0/1506945936 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 2 v2) v1 ==== 75+0+1093 (secure 0 0 0) 0x7fd12c05ade0 con 0x7fd1380ffc10 2024-05-04T12:15:04.205 INFO:teuthology.orchestra.run.smithi079.stdout:e2 2024-05-04T12:15:04.205 INFO:teuthology.orchestra.run.smithi079.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-05-04T12:15:04.205 INFO:teuthology.orchestra.run.smithi079.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-05-04T12:15:04.205 INFO:teuthology.orchestra.run.smithi079.stdout:legacy client fscid: 1 2024-05-04T12:15:04.206 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-05-04T12:15:04.206 INFO:teuthology.orchestra.run.smithi079.stdout:Filesystem 'cephfs' (1) 2024-05-04T12:15:04.206 INFO:teuthology.orchestra.run.smithi079.stdout:fs_name cephfs 2024-05-04T12:15:04.206 INFO:teuthology.orchestra.run.smithi079.stdout:epoch 2 2024-05-04T12:15:04.206 INFO:teuthology.orchestra.run.smithi079.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-05-04T12:15:04.206 INFO:teuthology.orchestra.run.smithi079.stdout:created 2024-05-04T12:15:00.083118+0000 2024-05-04T12:15:04.206 INFO:teuthology.orchestra.run.smithi079.stdout:modified 2024-05-04T12:15:00.083187+0000 2024-05-04T12:15:04.206 INFO:teuthology.orchestra.run.smithi079.stdout:tableserver 0 2024-05-04T12:15:04.206 INFO:teuthology.orchestra.run.smithi079.stdout:root 0 2024-05-04T12:15:04.206 INFO:teuthology.orchestra.run.smithi079.stdout:session_timeout 60 2024-05-04T12:15:04.206 INFO:teuthology.orchestra.run.smithi079.stdout:session_autoclose 300 2024-05-04T12:15:04.206 INFO:teuthology.orchestra.run.smithi079.stdout:max_file_size 1099511627776 2024-05-04T12:15:04.206 INFO:teuthology.orchestra.run.smithi079.stdout:required_client_features {} 2024-05-04T12:15:04.206 INFO:teuthology.orchestra.run.smithi079.stdout:last_failure 0 2024-05-04T12:15:04.206 INFO:teuthology.orchestra.run.smithi079.stdout:last_failure_osd_epoch 0 2024-05-04T12:15:04.207 INFO:teuthology.orchestra.run.smithi079.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-05-04T12:15:04.207 INFO:teuthology.orchestra.run.smithi079.stdout:max_mds 1 2024-05-04T12:15:04.207 INFO:teuthology.orchestra.run.smithi079.stdout:in 2024-05-04T12:15:04.207 INFO:teuthology.orchestra.run.smithi079.stdout:up {} 2024-05-04T12:15:04.207 INFO:teuthology.orchestra.run.smithi079.stdout:failed 2024-05-04T12:15:04.207 INFO:teuthology.orchestra.run.smithi079.stdout:damaged 2024-05-04T12:15:04.207 INFO:teuthology.orchestra.run.smithi079.stdout:stopped 2024-05-04T12:15:04.207 INFO:teuthology.orchestra.run.smithi079.stdout:data_pools [3] 2024-05-04T12:15:04.207 INFO:teuthology.orchestra.run.smithi079.stdout:metadata_pool 2 2024-05-04T12:15:04.207 INFO:teuthology.orchestra.run.smithi079.stdout:inline_data disabled 2024-05-04T12:15:04.207 INFO:teuthology.orchestra.run.smithi079.stdout:balancer 2024-05-04T12:15:04.207 INFO:teuthology.orchestra.run.smithi079.stdout:bal_rank_mask -1 2024-05-04T12:15:04.207 INFO:teuthology.orchestra.run.smithi079.stdout:standby_count_wanted 0 2024-05-04T12:15:04.207 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-05-04T12:15:04.208 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-05-04T12:15:04.208 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:04.208+0000 7fd13d0c5700 1 -- 172.21.15.79:0/1506945936 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7fd12006c4e0 msgr2=0x7fd12006e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:15:04.208 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:04.208+0000 7fd13d0c5700 1 --2- 172.21.15.79:0/1506945936 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7fd12006c4e0 0x7fd12006e9a0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fd128006fd0 tx=0x7fd128008040 comp rx=0 tx=0).stop 2024-05-04T12:15:04.208 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:04.208+0000 7fd13d0c5700 1 -- 172.21.15.79:0/1506945936 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd1380ffc10 msgr2=0x7fd138196530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:15:04.208 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:04.208+0000 7fd13d0c5700 1 --2- 172.21.15.79:0/1506945936 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd1380ffc10 0x7fd138196530 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fd12c00bda0 tx=0x7fd12c00be80 comp rx=0 tx=0).stop 2024-05-04T12:15:04.209 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:04.208+0000 7fd13d0c5700 1 -- 172.21.15.79:0/1506945936 shutdown_connections 2024-05-04T12:15:04.209 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:04.208+0000 7fd13d0c5700 1 --2- 172.21.15.79:0/1506945936 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7fd12006c4e0 0x7fd12006e9a0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:04.209 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:04.208+0000 7fd13d0c5700 1 --2- 172.21.15.79:0/1506945936 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd1380ffc10 0x7fd138196530 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:04.209 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:04.208+0000 7fd13d0c5700 1 --2- 172.21.15.79:0/1506945936 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd138100e10 0x7fd138196a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:04.209 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:04.208+0000 7fd13d0c5700 1 -- 172.21.15.79:0/1506945936 >> 172.21.15.79:0/1506945936 conn(0x7fd1380fb190 msgr2=0x7fd138068580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:15:04.209 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:04.208+0000 7fd13d0c5700 1 -- 172.21.15.79:0/1506945936 shutdown_connections 2024-05-04T12:15:04.209 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:04.208+0000 7fd13d0c5700 1 -- 172.21.15.79:0/1506945936 wait complete. 2024-05-04T12:15:04.209 INFO:teuthology.orchestra.run.smithi079.stderr:dumped fsmap epoch 2 2024-05-04T12:15:04.839 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-05-04T12:15:04.849 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi079.front.sepia.ceph.com 2024-05-04T12:15:04.849 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d9b985c4-0a0e-11ef-bc97-c7b262605968 -- bash -c 'ceph fs set cephfs max_mds 1' 2024-05-04T12:15:05.055 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/d9b985c4-0a0e-11ef-bc97-c7b262605968/mon.smithi079/config 2024-05-04T12:15:05.437 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:05 smithi079 ceph-mon[27222]: pgmap v179: 65 pgs: 50 active+clean, 9 creating+peering, 6 unknown; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-04T12:15:05.437 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:05 smithi079 ceph-mon[27222]: from='client.? 172.21.15.79:0/1506945936' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-05-04T12:15:05.438 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:05 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:15:05.438 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:05 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:15:05.438 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:05 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:15:05.438 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:05 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi146.tfajce", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-04T12:15:05.438 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:05 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi146.tfajce", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-04T12:15:05.438 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:05 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-04T12:15:05.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:05 smithi146 ceph-mon[35311]: pgmap v179: 65 pgs: 50 active+clean, 9 creating+peering, 6 unknown; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-04T12:15:05.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:05 smithi146 ceph-mon[35311]: from='client.? 172.21.15.79:0/1506945936' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-05-04T12:15:05.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:05 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:15:05.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:05 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:15:05.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:05 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:15:05.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:05 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi146.tfajce", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-04T12:15:05.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:05 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi146.tfajce", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-04T12:15:05.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:05 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-04T12:15:06.217 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:06 smithi146 ceph-mon[35311]: Deploying daemon mds.cephfs.smithi146.tfajce on smithi146 2024-05-04T12:15:06.217 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:06 smithi146 ceph-mon[35311]: mds.? [v2:172.21.15.79:6826/4053411185,v1:172.21.15.79:6827/4053411185] up:boot 2024-05-04T12:15:06.217 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:06 smithi146 ceph-mon[35311]: daemon mds.cephfs.smithi079.pspich assigned to filesystem cephfs as rank 0 (now has 1 ranks) 2024-05-04T12:15:06.217 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:06 smithi146 ceph-mon[35311]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-05-04T12:15:06.218 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:06 smithi146 ceph-mon[35311]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-05-04T12:15:06.218 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:06 smithi146 ceph-mon[35311]: Cluster is now healthy 2024-05-04T12:15:06.218 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:06 smithi146 ceph-mon[35311]: fsmap cephfs:0 1 up:standby 2024-05-04T12:15:06.218 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:06 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi079.pspich"}]: dispatch 2024-05-04T12:15:06.218 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:06 smithi146 ceph-mon[35311]: fsmap cephfs:1 {0=cephfs.smithi079.pspich=up:creating} 2024-05-04T12:15:06.218 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:06 smithi146 ceph-mon[35311]: daemon mds.cephfs.smithi079.pspich is now active in filesystem cephfs as rank 0 2024-05-04T12:15:06.403 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:06 smithi079 ceph-mon[27222]: Deploying daemon mds.cephfs.smithi146.tfajce on smithi146 2024-05-04T12:15:06.403 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:06 smithi079 ceph-mon[27222]: mds.? [v2:172.21.15.79:6826/4053411185,v1:172.21.15.79:6827/4053411185] up:boot 2024-05-04T12:15:06.403 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:06 smithi079 ceph-mon[27222]: daemon mds.cephfs.smithi079.pspich assigned to filesystem cephfs as rank 0 (now has 1 ranks) 2024-05-04T12:15:06.403 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:06 smithi079 ceph-mon[27222]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-05-04T12:15:06.403 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:06 smithi079 ceph-mon[27222]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-05-04T12:15:06.403 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:06 smithi079 ceph-mon[27222]: Cluster is now healthy 2024-05-04T12:15:06.403 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:06 smithi079 ceph-mon[27222]: fsmap cephfs:0 1 up:standby 2024-05-04T12:15:06.403 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:06 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi079.pspich"}]: dispatch 2024-05-04T12:15:06.403 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:06 smithi079 ceph-mon[27222]: fsmap cephfs:1 {0=cephfs.smithi079.pspich=up:creating} 2024-05-04T12:15:06.403 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:06 smithi079 ceph-mon[27222]: daemon mds.cephfs.smithi079.pspich is now active in filesystem cephfs as rank 0 2024-05-04T12:15:06.620 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:06.619+0000 7f6c32df3700 1 -- 172.21.15.79:0/294885966 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6c2c101eb0 msgr2=0x7f6c2c102310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:15:06.620 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:06.619+0000 7f6c32df3700 1 --2- 172.21.15.79:0/294885966 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6c2c101eb0 0x7f6c2c102310 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f6c1c009910 tx=0x7f6c1c009c20 comp rx=0 tx=0).stop 2024-05-04T12:15:06.620 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:06.619+0000 7f6c32df3700 1 -- 172.21.15.79:0/294885966 shutdown_connections 2024-05-04T12:15:06.620 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:06.619+0000 7f6c32df3700 1 --2- 172.21.15.79:0/294885966 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6c2c101eb0 0x7f6c2c102310 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:06.620 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:06.619+0000 7f6c32df3700 1 --2- 172.21.15.79:0/294885966 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6c2c100cb0 0x7f6c2c1010d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:06.620 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:06.619+0000 7f6c32df3700 1 -- 172.21.15.79:0/294885966 >> 172.21.15.79:0/294885966 conn(0x7f6c2c0fc230 msgr2=0x7f6c2c0fe690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:15:06.620 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:06.619+0000 7f6c32df3700 1 -- 172.21.15.79:0/294885966 shutdown_connections 2024-05-04T12:15:06.620 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:06.619+0000 7f6c32df3700 1 -- 172.21.15.79:0/294885966 wait complete. 2024-05-04T12:15:06.620 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:06.620+0000 7f6c32df3700 1 Processor -- start 2024-05-04T12:15:06.620 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:06.620+0000 7f6c32df3700 1 -- start start 2024-05-04T12:15:06.621 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:06.620+0000 7f6c32df3700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6c2c100cb0 0x7f6c2c196550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:15:06.621 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:06.620+0000 7f6c32df3700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6c2c101eb0 0x7f6c2c196a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:15:06.621 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:06.620+0000 7f6c32df3700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c2c1970b0 con 0x7f6c2c100cb0 2024-05-04T12:15:06.621 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:06.620+0000 7f6c32df3700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c2c1971f0 con 0x7f6c2c101eb0 2024-05-04T12:15:06.621 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:06.620+0000 7f6c30b8f700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6c2c100cb0 0x7f6c2c196550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:15:06.621 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:06.620+0000 7f6c30b8f700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6c2c100cb0 0x7f6c2c196550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:56674/0 (socket says 172.21.15.79:56674) 2024-05-04T12:15:06.621 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:06.620+0000 7f6c30b8f700 1 -- 172.21.15.79:0/1132485018 learned_addr learned my addr 172.21.15.79:0/1132485018 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:15:06.621 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:06.620+0000 7f6c2bfff700 1 --2- 172.21.15.79:0/1132485018 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6c2c101eb0 0x7f6c2c196a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:15:06.621 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:06.620+0000 7f6c30b8f700 1 -- 172.21.15.79:0/1132485018 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6c2c101eb0 msgr2=0x7f6c2c196a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:15:06.621 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:06.620+0000 7f6c30b8f700 1 --2- 172.21.15.79:0/1132485018 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6c2c101eb0 0x7f6c2c196a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:06.621 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:06.620+0000 7f6c30b8f700 1 -- 172.21.15.79:0/1132485018 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c1c0095f0 con 0x7f6c2c100cb0 2024-05-04T12:15:06.621 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:06.620+0000 7f6c2bfff700 1 --2- 172.21.15.79:0/1132485018 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6c2c101eb0 0x7f6c2c196a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-04T12:15:06.621 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:06.620+0000 7f6c30b8f700 1 --2- 172.21.15.79:0/1132485018 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6c2c100cb0 0x7f6c2c196550 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f6c2400b9c0 tx=0x7f6c2400bd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:15:06.621 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:06.620+0000 7f6c29ffb700 1 -- 172.21.15.79:0/1132485018 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6c2400c980 con 0x7f6c2c100cb0 2024-05-04T12:15:06.621 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:06.621+0000 7f6c32df3700 1 -- 172.21.15.79:0/1132485018 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c2c19bca0 con 0x7f6c2c100cb0 2024-05-04T12:15:06.622 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:06.621+0000 7f6c29ffb700 1 -- 172.21.15.79:0/1132485018 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f6c24014440 con 0x7f6c2c100cb0 2024-05-04T12:15:06.622 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:06.621+0000 7f6c29ffb700 1 -- 172.21.15.79:0/1132485018 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6c24012550 con 0x7f6c2c100cb0 2024-05-04T12:15:06.622 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:06.621+0000 7f6c32df3700 1 -- 172.21.15.79:0/1132485018 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c2c19c1a0 con 0x7f6c2c100cb0 2024-05-04T12:15:06.623 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:06.622+0000 7f6c32df3700 1 -- 172.21.15.79:0/1132485018 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c2c1098b0 con 0x7f6c2c100cb0 2024-05-04T12:15:06.623 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:06.622+0000 7f6c29ffb700 1 -- 172.21.15.79:0/1132485018 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f6c240145b0 con 0x7f6c2c100cb0 2024-05-04T12:15:06.623 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:06.622+0000 7f6c29ffb700 1 --2- 172.21.15.79:0/1132485018 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f6c1806c4e0 0x7f6c1806e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:15:06.623 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:06.622+0000 7f6c29ffb700 1 -- 172.21.15.79:0/1132485018 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5468+0+0 (secure 0 0 0) 0x7f6c24006e40 con 0x7f6c2c100cb0 2024-05-04T12:15:06.623 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:06.622+0000 7f6c2bfff700 1 --2- 172.21.15.79:0/1132485018 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f6c1806c4e0 0x7f6c1806e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:15:06.623 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:06.623+0000 7f6c2bfff700 1 --2- 172.21.15.79:0/1132485018 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f6c1806c4e0 0x7f6c1806e9a0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f6c1c0059d0 tx=0x7f6c1c005920 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:15:06.626 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:06.627+0000 7f6c29ffb700 1 -- 172.21.15.79:0/1132485018 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f6c24059270 con 0x7f6c2c100cb0 2024-05-04T12:15:06.859 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:06.860+0000 7f6c32df3700 1 -- 172.21.15.79:0/1132485018 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"} v 0) v1 -- 0x7f6c2c19c870 con 0x7f6c2c100cb0 2024-05-04T12:15:07.106 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:07.106+0000 7f6c29ffb700 1 -- 172.21.15.79:0/1132485018 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]=0 v6) v1 ==== 105+0+0 (secure 0 0 0) 0x7f6c24058e00 con 0x7f6c2c100cb0 2024-05-04T12:15:07.109 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:07.109+0000 7f6c32df3700 1 -- 172.21.15.79:0/1132485018 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f6c1806c4e0 msgr2=0x7f6c1806e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:15:07.109 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:07.109+0000 7f6c32df3700 1 --2- 172.21.15.79:0/1132485018 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f6c1806c4e0 0x7f6c1806e9a0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f6c1c0059d0 tx=0x7f6c1c005920 comp rx=0 tx=0).stop 2024-05-04T12:15:07.109 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:07.109+0000 7f6c32df3700 1 -- 172.21.15.79:0/1132485018 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6c2c100cb0 msgr2=0x7f6c2c196550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:15:07.109 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:07.109+0000 7f6c32df3700 1 --2- 172.21.15.79:0/1132485018 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6c2c100cb0 0x7f6c2c196550 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f6c2400b9c0 tx=0x7f6c2400bd80 comp rx=0 tx=0).stop 2024-05-04T12:15:07.109 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:07.109+0000 7f6c32df3700 1 -- 172.21.15.79:0/1132485018 shutdown_connections 2024-05-04T12:15:07.109 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:07.109+0000 7f6c32df3700 1 --2- 172.21.15.79:0/1132485018 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f6c1806c4e0 0x7f6c1806e9a0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:07.109 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:07.109+0000 7f6c32df3700 1 --2- 172.21.15.79:0/1132485018 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6c2c100cb0 0x7f6c2c196550 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:07.109 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:07.109+0000 7f6c32df3700 1 --2- 172.21.15.79:0/1132485018 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6c2c101eb0 0x7f6c2c196a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:07.109 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:07.109+0000 7f6c32df3700 1 -- 172.21.15.79:0/1132485018 >> 172.21.15.79:0/1132485018 conn(0x7f6c2c0fc230 msgr2=0x7f6c2c1050e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:15:07.109 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:07.109+0000 7f6c32df3700 1 -- 172.21.15.79:0/1132485018 shutdown_connections 2024-05-04T12:15:07.109 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:07.109+0000 7f6c32df3700 1 -- 172.21.15.79:0/1132485018 wait complete. 2024-05-04T12:15:07.120 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:07 smithi079 ceph-mon[27222]: pgmap v180: 65 pgs: 59 active+clean, 6 creating+peering; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-04T12:15:07.120 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:07 smithi079 ceph-mon[27222]: mds.? [v2:172.21.15.79:6826/4053411185,v1:172.21.15.79:6827/4053411185] up:active 2024-05-04T12:15:07.120 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:07 smithi079 ceph-mon[27222]: fsmap cephfs:1 {0=cephfs.smithi079.pspich=up:active} 2024-05-04T12:15:07.120 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:07 smithi079 ceph-mon[27222]: from='client.? 172.21.15.79:0/1132485018' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]: dispatch 2024-05-04T12:15:07.121 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:07 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:15:07.121 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:07 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:15:07.121 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:07 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:15:07.121 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:07 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi079.hzcmip", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-04T12:15:07.121 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:07 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi079.hzcmip", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-04T12:15:07.121 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:07 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-04T12:15:07.277 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:07 smithi146 ceph-mon[35311]: pgmap v180: 65 pgs: 59 active+clean, 6 creating+peering; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-04T12:15:07.277 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:07 smithi146 ceph-mon[35311]: mds.? [v2:172.21.15.79:6826/4053411185,v1:172.21.15.79:6827/4053411185] up:active 2024-05-04T12:15:07.277 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:07 smithi146 ceph-mon[35311]: fsmap cephfs:1 {0=cephfs.smithi079.pspich=up:active} 2024-05-04T12:15:07.277 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:07 smithi146 ceph-mon[35311]: from='client.? 172.21.15.79:0/1132485018' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]: dispatch 2024-05-04T12:15:07.277 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:07 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:15:07.277 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:07 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:15:07.278 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:07 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:15:07.278 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:07 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi079.hzcmip", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-04T12:15:07.278 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:07 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi079.hzcmip", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-04T12:15:07.278 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:07 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-04T12:15:07.703 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-05-04T12:15:07.714 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi079.front.sepia.ceph.com 2024-05-04T12:15:07.714 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d9b985c4-0a0e-11ef-bc97-c7b262605968 -- bash -c 'ceph fs set cephfs allow_standby_replay false' 2024-05-04T12:15:08.277 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:08 smithi079 ceph-mon[27222]: Deploying daemon mds.cephfs.smithi079.hzcmip on smithi079 2024-05-04T12:15:08.277 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:08 smithi079 ceph-mon[27222]: from='client.? 172.21.15.79:0/1132485018' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]': finished 2024-05-04T12:15:08.277 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:08 smithi079 ceph-mon[27222]: mds.? [v2:172.21.15.146:6824/4162809119,v1:172.21.15.146:6825/4162809119] up:boot 2024-05-04T12:15:08.277 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:08 smithi079 ceph-mon[27222]: fsmap cephfs:1 {0=cephfs.smithi079.pspich=up:active} 1 up:standby 2024-05-04T12:15:08.277 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:08 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi146.tfajce"}]: dispatch 2024-05-04T12:15:08.277 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:08 smithi079 ceph-mon[27222]: fsmap cephfs:1 {0=cephfs.smithi079.pspich=up:active} 1 up:standby 2024-05-04T12:15:08.442 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/d9b985c4-0a0e-11ef-bc97-c7b262605968/mon.smithi079/config 2024-05-04T12:15:08.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:08 smithi146 ceph-mon[35311]: Deploying daemon mds.cephfs.smithi079.hzcmip on smithi079 2024-05-04T12:15:08.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:08 smithi146 ceph-mon[35311]: from='client.? 172.21.15.79:0/1132485018' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]': finished 2024-05-04T12:15:08.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:08 smithi146 ceph-mon[35311]: mds.? [v2:172.21.15.146:6824/4162809119,v1:172.21.15.146:6825/4162809119] up:boot 2024-05-04T12:15:08.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:08 smithi146 ceph-mon[35311]: fsmap cephfs:1 {0=cephfs.smithi079.pspich=up:active} 1 up:standby 2024-05-04T12:15:08.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:08 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi146.tfajce"}]: dispatch 2024-05-04T12:15:08.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:08 smithi146 ceph-mon[35311]: fsmap cephfs:1 {0=cephfs.smithi079.pspich=up:active} 1 up:standby 2024-05-04T12:15:09.270 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:09 smithi146 ceph-mon[35311]: pgmap v181: 65 pgs: 65 active+clean; 578 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 814 B/s wr, 4 op/s 2024-05-04T12:15:09.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:09 smithi079 ceph-mon[27222]: pgmap v181: 65 pgs: 65 active+clean; 578 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 814 B/s wr, 4 op/s 2024-05-04T12:15:10.625 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:10.624+0000 7fc475380700 1 -- 172.21.15.79:0/1848791854 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc470100e10 msgr2=0x7fc470101270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:15:10.625 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:10.624+0000 7fc475380700 1 --2- 172.21.15.79:0/1848791854 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc470100e10 0x7fc470101270 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fc458009910 tx=0x7fc458009c20 comp rx=0 tx=0).stop 2024-05-04T12:15:10.625 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:10.624+0000 7fc475380700 1 -- 172.21.15.79:0/1848791854 shutdown_connections 2024-05-04T12:15:10.625 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:10.624+0000 7fc475380700 1 --2- 172.21.15.79:0/1848791854 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc470100e10 0x7fc470101270 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:10.626 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:10.624+0000 7fc475380700 1 --2- 172.21.15.79:0/1848791854 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc4700ffc10 0x7fc470100030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:10.626 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:10.624+0000 7fc475380700 1 -- 172.21.15.79:0/1848791854 >> 172.21.15.79:0/1848791854 conn(0x7fc4700fb190 msgr2=0x7fc4700fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:15:10.626 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:10.624+0000 7fc475380700 1 -- 172.21.15.79:0/1848791854 shutdown_connections 2024-05-04T12:15:10.626 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:10.624+0000 7fc475380700 1 -- 172.21.15.79:0/1848791854 wait complete. 2024-05-04T12:15:10.626 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:10.625+0000 7fc475380700 1 Processor -- start 2024-05-04T12:15:10.626 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:10.625+0000 7fc475380700 1 -- start start 2024-05-04T12:15:10.626 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:10.625+0000 7fc475380700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc4700ffc10 0x7fc470196480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:15:10.626 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:10.625+0000 7fc475380700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc470100e10 0x7fc4701969c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:15:10.626 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:10.625+0000 7fc475380700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc470196fe0 con 0x7fc4700ffc10 2024-05-04T12:15:10.626 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:10.625+0000 7fc475380700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc470197120 con 0x7fc470100e10 2024-05-04T12:15:10.626 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:10.625+0000 7fc46effd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc4700ffc10 0x7fc470196480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:15:10.627 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:10.625+0000 7fc46effd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc4700ffc10 0x7fc470196480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:56692/0 (socket says 172.21.15.79:56692) 2024-05-04T12:15:10.627 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:10.625+0000 7fc46effd700 1 -- 172.21.15.79:0/4321136 learned_addr learned my addr 172.21.15.79:0/4321136 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:15:10.627 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:10.625+0000 7fc4667fc700 1 --2- 172.21.15.79:0/4321136 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc470100e10 0x7fc4701969c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:15:10.627 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:10.625+0000 7fc46effd700 1 -- 172.21.15.79:0/4321136 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc470100e10 msgr2=0x7fc4701969c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:15:10.627 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:10.625+0000 7fc46effd700 1 --2- 172.21.15.79:0/4321136 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc470100e10 0x7fc4701969c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:10.627 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:10.625+0000 7fc46effd700 1 -- 172.21.15.79:0/4321136 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4580095f0 con 0x7fc4700ffc10 2024-05-04T12:15:10.627 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:10.625+0000 7fc46effd700 1 --2- 172.21.15.79:0/4321136 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc4700ffc10 0x7fc470196480 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fc46000d900 tx=0x7fc46000dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:15:10.627 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:10.626+0000 7fc46cff9700 1 -- 172.21.15.79:0/4321136 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc4600041f0 con 0x7fc4700ffc10 2024-05-04T12:15:10.627 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:10.626+0000 7fc46cff9700 1 -- 172.21.15.79:0/4321136 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fc460004350 con 0x7fc4700ffc10 2024-05-04T12:15:10.627 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:10.626+0000 7fc46cff9700 1 -- 172.21.15.79:0/4321136 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc460003d20 con 0x7fc4700ffc10 2024-05-04T12:15:10.627 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:10.626+0000 7fc475380700 1 -- 172.21.15.79:0/4321136 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc47019bbd0 con 0x7fc4700ffc10 2024-05-04T12:15:10.627 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:10.626+0000 7fc475380700 1 -- 172.21.15.79:0/4321136 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc47019c120 con 0x7fc4700ffc10 2024-05-04T12:15:10.628 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:10.627+0000 7fc475380700 1 -- 172.21.15.79:0/4321136 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc470108700 con 0x7fc4700ffc10 2024-05-04T12:15:10.628 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:10.627+0000 7fc46cff9700 1 -- 172.21.15.79:0/4321136 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7fc460009730 con 0x7fc4700ffc10 2024-05-04T12:15:10.628 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:10.627+0000 7fc46cff9700 1 --2- 172.21.15.79:0/4321136 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7fc45c06c4e0 0x7fc45c06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:15:10.628 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:10.628+0000 7fc46cff9700 1 -- 172.21.15.79:0/4321136 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5468+0+0 (secure 0 0 0) 0x7fc46008b6b0 con 0x7fc4700ffc10 2024-05-04T12:15:10.628 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:10.628+0000 7fc4667fc700 1 --2- 172.21.15.79:0/4321136 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7fc45c06c4e0 0x7fc45c06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:15:10.628 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:10.628+0000 7fc4667fc700 1 --2- 172.21.15.79:0/4321136 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7fc45c06c4e0 0x7fc45c06e9a0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fc458006010 tx=0x7fc45800b540 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:15:10.631 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:10.632+0000 7fc46cff9700 1 -- 172.21.15.79:0/4321136 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fc460059920 con 0x7fc4700ffc10 2024-05-04T12:15:10.873 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:10.874+0000 7fc475380700 1 -- 172.21.15.79:0/4321136 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fc47019c8c0 con 0x7fc4700ffc10 2024-05-04T12:15:11.118 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:11.119+0000 7fc46cff9700 1 -- 172.21.15.79:0/4321136 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v8) v1 ==== 122+0+0 (secure 0 0 0) 0x7fc460003e80 con 0x7fc4700ffc10 2024-05-04T12:15:11.121 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:11.121+0000 7fc475380700 1 -- 172.21.15.79:0/4321136 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7fc45c06c4e0 msgr2=0x7fc45c06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:15:11.121 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:11.121+0000 7fc475380700 1 --2- 172.21.15.79:0/4321136 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7fc45c06c4e0 0x7fc45c06e9a0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fc458006010 tx=0x7fc45800b540 comp rx=0 tx=0).stop 2024-05-04T12:15:11.121 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:11.121+0000 7fc475380700 1 -- 172.21.15.79:0/4321136 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc4700ffc10 msgr2=0x7fc470196480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:15:11.121 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:11.121+0000 7fc475380700 1 --2- 172.21.15.79:0/4321136 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc4700ffc10 0x7fc470196480 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fc46000d900 tx=0x7fc46000dc10 comp rx=0 tx=0).stop 2024-05-04T12:15:11.121 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:11.121+0000 7fc475380700 1 -- 172.21.15.79:0/4321136 shutdown_connections 2024-05-04T12:15:11.121 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:11.121+0000 7fc475380700 1 --2- 172.21.15.79:0/4321136 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7fc45c06c4e0 0x7fc45c06e9a0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:11.121 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:11.121+0000 7fc475380700 1 --2- 172.21.15.79:0/4321136 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc4700ffc10 0x7fc470196480 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:11.121 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:11.121+0000 7fc475380700 1 --2- 172.21.15.79:0/4321136 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc470100e10 0x7fc4701969c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:11.121 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:11.121+0000 7fc475380700 1 -- 172.21.15.79:0/4321136 >> 172.21.15.79:0/4321136 conn(0x7fc4700fb190 msgr2=0x7fc470068580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:15:11.121 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:11.122+0000 7fc475380700 1 -- 172.21.15.79:0/4321136 shutdown_connections 2024-05-04T12:15:11.121 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:11.122+0000 7fc475380700 1 -- 172.21.15.79:0/4321136 wait complete. 2024-05-04T12:15:11.380 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:11 smithi146 ceph-mon[35311]: pgmap v182: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.6 KiB/s wr, 5 op/s 2024-05-04T12:15:11.380 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:11 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:15:11.380 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:11 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:15:11.380 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:11 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:15:11.380 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:11 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi146.mkcmzm", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-04T12:15:11.380 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:11 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi146.mkcmzm", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-04T12:15:11.380 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:11 smithi146 ceph-mon[35311]: from='client.? 172.21.15.79:0/4321136' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]: dispatch 2024-05-04T12:15:11.380 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:11 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-04T12:15:11.431 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:11 smithi079 ceph-mon[27222]: pgmap v182: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.6 KiB/s wr, 5 op/s 2024-05-04T12:15:11.431 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:11 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:15:11.431 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:11 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:15:11.431 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:11 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:15:11.431 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:11 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi146.mkcmzm", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-04T12:15:11.431 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:11 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi146.mkcmzm", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-04T12:15:11.431 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:11 smithi079 ceph-mon[27222]: from='client.? 172.21.15.79:0/4321136' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]: dispatch 2024-05-04T12:15:11.431 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:11 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-04T12:15:11.761 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-05-04T12:15:11.771 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi079.front.sepia.ceph.com 2024-05-04T12:15:11.771 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d9b985c4-0a0e-11ef-bc97-c7b262605968 -- bash -c 'ceph fs set cephfs inline_data true --yes-i-really-really-mean-it' 2024-05-04T12:15:11.979 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/d9b985c4-0a0e-11ef-bc97-c7b262605968/mon.smithi079/config 2024-05-04T12:15:12.359 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:12 smithi079 ceph-mon[27222]: Deploying daemon mds.cephfs.smithi146.mkcmzm on smithi146 2024-05-04T12:15:12.359 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:12 smithi079 ceph-mon[27222]: mds.? [v2:172.21.15.79:6828/4287465339,v1:172.21.15.79:6829/4287465339] up:boot 2024-05-04T12:15:12.359 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:12 smithi079 ceph-mon[27222]: from='client.? 172.21.15.79:0/4321136' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]': finished 2024-05-04T12:15:12.359 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:12 smithi079 ceph-mon[27222]: fsmap cephfs:1 {0=cephfs.smithi079.pspich=up:active} 2 up:standby 2024-05-04T12:15:12.359 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:12 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi079.hzcmip"}]: dispatch 2024-05-04T12:15:12.359 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:12 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-04T12:15:12.359 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:12 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:15:12.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:12 smithi146 ceph-mon[35311]: Deploying daemon mds.cephfs.smithi146.mkcmzm on smithi146 2024-05-04T12:15:12.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:12 smithi146 ceph-mon[35311]: mds.? [v2:172.21.15.79:6828/4287465339,v1:172.21.15.79:6829/4287465339] up:boot 2024-05-04T12:15:12.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:12 smithi146 ceph-mon[35311]: from='client.? 172.21.15.79:0/4321136' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]': finished 2024-05-04T12:15:12.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:12 smithi146 ceph-mon[35311]: fsmap cephfs:1 {0=cephfs.smithi079.pspich=up:active} 2 up:standby 2024-05-04T12:15:12.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:12 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi079.hzcmip"}]: dispatch 2024-05-04T12:15:12.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:12 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-04T12:15:12.384 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:12 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:15:13.364 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:13 smithi079 ceph-mon[27222]: pgmap v183: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.4 KiB/s wr, 4 op/s 2024-05-04T12:15:13.429 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:13 smithi146 ceph-mon[35311]: pgmap v183: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.4 KiB/s wr, 4 op/s 2024-05-04T12:15:13.596 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:13.597+0000 7f6dcd09f700 1 -- 172.21.15.79:0/1287915537 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6dc80ffbc0 msgr2=0x7f6dc80fffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:15:13.596 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:13.597+0000 7f6dcd09f700 1 --2- 172.21.15.79:0/1287915537 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6dc80ffbc0 0x7f6dc80fffe0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f6db8009910 tx=0x7f6db8009c20 comp rx=0 tx=0).stop 2024-05-04T12:15:13.596 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:13.597+0000 7f6dcd09f700 1 -- 172.21.15.79:0/1287915537 shutdown_connections 2024-05-04T12:15:13.596 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:13.597+0000 7f6dcd09f700 1 --2- 172.21.15.79:0/1287915537 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6dc8100dc0 0x7f6dc8101220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:13.597 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:13.597+0000 7f6dcd09f700 1 --2- 172.21.15.79:0/1287915537 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6dc80ffbc0 0x7f6dc80fffe0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:13.597 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:13.597+0000 7f6dcd09f700 1 -- 172.21.15.79:0/1287915537 >> 172.21.15.79:0/1287915537 conn(0x7f6dc80fb160 msgr2=0x7f6dc80fd5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:15:13.597 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:13.597+0000 7f6dcd09f700 1 -- 172.21.15.79:0/1287915537 shutdown_connections 2024-05-04T12:15:13.597 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:13.597+0000 7f6dcd09f700 1 -- 172.21.15.79:0/1287915537 wait complete. 2024-05-04T12:15:13.598 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:13.598+0000 7f6dcd09f700 1 Processor -- start 2024-05-04T12:15:13.598 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:13.598+0000 7f6dcd09f700 1 -- start start 2024-05-04T12:15:13.598 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:13.598+0000 7f6dcd09f700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6dc80ffbc0 0x7f6dc8196800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:15:13.598 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:13.598+0000 7f6dcd09f700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6dc8100dc0 0x7f6dc8196d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:15:13.598 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:13.598+0000 7f6dcd09f700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dc81972d0 con 0x7f6dc8100dc0 2024-05-04T12:15:13.598 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:13.598+0000 7f6dcd09f700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dc8197410 con 0x7f6dc80ffbc0 2024-05-04T12:15:13.598 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:13.598+0000 7f6dc659c700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6dc8100dc0 0x7f6dc8196d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:15:13.599 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:13.598+0000 7f6dc6d9d700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6dc80ffbc0 0x7f6dc8196800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:15:13.599 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:13.598+0000 7f6dc659c700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6dc8100dc0 0x7f6dc8196d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:35778/0 (socket says 172.21.15.79:35778) 2024-05-04T12:15:13.599 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:13.598+0000 7f6dc659c700 1 -- 172.21.15.79:0/1634755410 learned_addr learned my addr 172.21.15.79:0/1634755410 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:15:13.599 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:13.598+0000 7f6dc659c700 1 -- 172.21.15.79:0/1634755410 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6dc80ffbc0 msgr2=0x7f6dc8196800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:15:13.599 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:13.598+0000 7f6dc659c700 1 --2- 172.21.15.79:0/1634755410 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6dc80ffbc0 0x7f6dc8196800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:13.599 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:13.598+0000 7f6dc659c700 1 -- 172.21.15.79:0/1634755410 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6db80095f0 con 0x7f6dc8100dc0 2024-05-04T12:15:13.599 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:13.599+0000 7f6dc659c700 1 --2- 172.21.15.79:0/1634755410 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6dc8100dc0 0x7f6dc8196d40 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f6dbc00c930 tx=0x7f6dbc00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:15:13.599 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:13.599+0000 7f6daffff700 1 -- 172.21.15.79:0/1634755410 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6dbc007ad0 con 0x7f6dc8100dc0 2024-05-04T12:15:13.599 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:13.599+0000 7f6dcd09f700 1 -- 172.21.15.79:0/1634755410 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6dc819bed0 con 0x7f6dc8100dc0 2024-05-04T12:15:13.599 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:13.599+0000 7f6daffff700 1 -- 172.21.15.79:0/1634755410 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f6dbc00ce80 con 0x7f6dc8100dc0 2024-05-04T12:15:13.599 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:13.599+0000 7f6daffff700 1 -- 172.21.15.79:0/1634755410 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6dbc018630 con 0x7f6dc8100dc0 2024-05-04T12:15:13.599 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:13.599+0000 7f6dcd09f700 1 -- 172.21.15.79:0/1634755410 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6dc819c3f0 con 0x7f6dc8100dc0 2024-05-04T12:15:13.600 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:13.600+0000 7f6dcd09f700 1 -- 172.21.15.79:0/1634755410 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6dc8068af0 con 0x7f6dc8100dc0 2024-05-04T12:15:13.600 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:13.600+0000 7f6daffff700 1 -- 172.21.15.79:0/1634755410 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f6dbc007c30 con 0x7f6dc8100dc0 2024-05-04T12:15:13.600 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:13.600+0000 7f6daffff700 1 --2- 172.21.15.79:0/1634755410 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f6db006c4e0 0x7f6db006e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:15:13.600 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:13.601+0000 7f6daffff700 1 -- 172.21.15.79:0/1634755410 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5468+0+0 (secure 0 0 0) 0x7f6dbc08b7b0 con 0x7f6dc8100dc0 2024-05-04T12:15:13.600 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:13.601+0000 7f6dc6d9d700 1 --2- 172.21.15.79:0/1634755410 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f6db006c4e0 0x7f6db006e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:15:13.600 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:13.601+0000 7f6dc6d9d700 1 --2- 172.21.15.79:0/1634755410 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f6db006c4e0 0x7f6db006e9a0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f6db8006010 tx=0x7f6db8005280 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:15:13.604 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:13.605+0000 7f6daffff700 1 -- 172.21.15.79:0/1634755410 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f6dbc059a20 con 0x7f6dc8100dc0 2024-05-04T12:15:13.852 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:13.853+0000 7f6dcd09f700 1 -- 172.21.15.79:0/1634755410 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f6dc819cb00 con 0x7f6dc8100dc0 2024-05-04T12:15:14.134 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:14.135+0000 7f6daffff700 1 -- 172.21.15.79:0/1634755410 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]=0 inline data enabled v9) v1 ==== 168+0+0 (secure 0 0 0) 0x7f6dbc0595b0 con 0x7f6dc8100dc0 2024-05-04T12:15:14.137 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:14.138+0000 7f6dcd09f700 1 -- 172.21.15.79:0/1634755410 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f6db006c4e0 msgr2=0x7f6db006e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:15:14.138 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:14.138+0000 7f6dcd09f700 1 --2- 172.21.15.79:0/1634755410 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f6db006c4e0 0x7f6db006e9a0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f6db8006010 tx=0x7f6db8005280 comp rx=0 tx=0).stop 2024-05-04T12:15:14.138 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:14.138+0000 7f6dcd09f700 1 -- 172.21.15.79:0/1634755410 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6dc8100dc0 msgr2=0x7f6dc8196d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:15:14.138 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:14.138+0000 7f6dcd09f700 1 --2- 172.21.15.79:0/1634755410 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6dc8100dc0 0x7f6dc8196d40 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f6dbc00c930 tx=0x7f6dbc00cc40 comp rx=0 tx=0).stop 2024-05-04T12:15:14.138 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:14.138+0000 7f6dcd09f700 1 -- 172.21.15.79:0/1634755410 shutdown_connections 2024-05-04T12:15:14.138 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:14.138+0000 7f6dcd09f700 1 --2- 172.21.15.79:0/1634755410 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f6db006c4e0 0x7f6db006e9a0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:14.138 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:14.138+0000 7f6dcd09f700 1 --2- 172.21.15.79:0/1634755410 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6dc80ffbc0 0x7f6dc8196800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:14.138 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:14.138+0000 7f6dcd09f700 1 --2- 172.21.15.79:0/1634755410 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6dc8100dc0 0x7f6dc8196d40 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:14.138 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:14.138+0000 7f6dcd09f700 1 -- 172.21.15.79:0/1634755410 >> 172.21.15.79:0/1634755410 conn(0x7f6dc80fb160 msgr2=0x7f6dc8068eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:15:14.138 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:14.138+0000 7f6dcd09f700 1 -- 172.21.15.79:0/1634755410 shutdown_connections 2024-05-04T12:15:14.138 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:14.138+0000 7f6dcd09f700 1 -- 172.21.15.79:0/1634755410 wait complete. 2024-05-04T12:15:14.139 INFO:teuthology.orchestra.run.smithi079.stderr:inline data enabled 2024-05-04T12:15:14.777 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:14 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:15:14.777 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:14 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:15:14.777 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:14 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:15:14.777 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:14 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:15:14.777 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:14 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:15:14.777 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:14 smithi146 ceph-mon[35311]: pgmap v184: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s wr, 3 op/s 2024-05-04T12:15:14.777 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:14 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-04T12:15:14.777 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:14 smithi146 ceph-mon[35311]: from='client.? 172.21.15.79:0/1634755410' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-04T12:15:14.777 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:14 smithi146 ceph-mon[35311]: Health check failed: 1 filesystem with deprecated feature inline_data (FS_INLINE_DATA_DEPRECATED) 2024-05-04T12:15:14.778 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:14 smithi146 ceph-mon[35311]: mds.? [v2:172.21.15.146:6826/178032877,v1:172.21.15.146:6827/178032877] up:boot 2024-05-04T12:15:14.778 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:14 smithi146 ceph-mon[35311]: from='client.? 172.21.15.79:0/1634755410' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]': finished 2024-05-04T12:15:14.778 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:14 smithi146 ceph-mon[35311]: fsmap cephfs:1 {0=cephfs.smithi079.pspich=up:active} 3 up:standby 2024-05-04T12:15:14.778 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:14 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi146.mkcmzm"}]: dispatch 2024-05-04T12:15:14.828 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:14 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:15:14.828 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:14 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:15:14.828 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:14 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:15:14.828 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:14 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:15:14.828 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:14 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:15:14.828 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:14 smithi079 ceph-mon[27222]: pgmap v184: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s wr, 3 op/s 2024-05-04T12:15:14.829 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:14 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-04T12:15:14.829 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:14 smithi079 ceph-mon[27222]: from='client.? 172.21.15.79:0/1634755410' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-04T12:15:14.829 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:14 smithi079 ceph-mon[27222]: Health check failed: 1 filesystem with deprecated feature inline_data (FS_INLINE_DATA_DEPRECATED) 2024-05-04T12:15:14.829 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:14 smithi079 ceph-mon[27222]: mds.? [v2:172.21.15.146:6826/178032877,v1:172.21.15.146:6827/178032877] up:boot 2024-05-04T12:15:14.829 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:14 smithi079 ceph-mon[27222]: from='client.? 172.21.15.79:0/1634755410' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]': finished 2024-05-04T12:15:14.829 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:14 smithi079 ceph-mon[27222]: fsmap cephfs:1 {0=cephfs.smithi079.pspich=up:active} 3 up:standby 2024-05-04T12:15:14.829 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:14 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi146.mkcmzm"}]: dispatch 2024-05-04T12:15:14.850 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-05-04T12:15:14.860 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi079.front.sepia.ceph.com 2024-05-04T12:15:14.860 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d9b985c4-0a0e-11ef-bc97-c7b262605968 -- bash -c 'ceph fs dump' 2024-05-04T12:15:15.536 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/d9b985c4-0a0e-11ef-bc97-c7b262605968/mon.smithi079/config 2024-05-04T12:15:16.864 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:16 smithi079 ceph-mon[27222]: mds.? [v2:172.21.15.79:6828/4287465339,v1:172.21.15.79:6829/4287465339] up:standby 2024-05-04T12:15:16.864 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:16 smithi079 ceph-mon[27222]: mds.? [v2:172.21.15.146:6824/4162809119,v1:172.21.15.146:6825/4162809119] up:standby 2024-05-04T12:15:16.864 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:16 smithi079 ceph-mon[27222]: Dropping low affinity active daemon mds.cephfs.smithi079.pspich in favor of higher affinity standby. 2024-05-04T12:15:16.864 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:16 smithi079 ceph-mon[27222]: Replacing daemon mds.cephfs.smithi079.pspich as rank 0 with standby daemon mds.cephfs.smithi079.hzcmip 2024-05-04T12:15:16.864 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:16 smithi079 ceph-mon[27222]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-05-04T12:15:16.864 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:16 smithi079 ceph-mon[27222]: fsmap cephfs:1 {0=cephfs.smithi079.pspich=up:active} 3 up:standby 2024-05-04T12:15:16.865 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:16 smithi079 ceph-mon[27222]: pgmap v186: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.4 KiB/s wr, 4 op/s 2024-05-04T12:15:16.865 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:16 smithi079 ceph-mon[27222]: osdmap e46: 6 total, 6 up, 6 in 2024-05-04T12:15:16.865 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:16 smithi079 ceph-mon[27222]: fsmap cephfs:1/1 {0=cephfs.smithi079.hzcmip=up:replay} 2 up:standby 2024-05-04T12:15:16.865 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:16 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:15:16.865 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:16 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:15:17.027 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:16 smithi146 ceph-mon[35311]: mds.? [v2:172.21.15.79:6828/4287465339,v1:172.21.15.79:6829/4287465339] up:standby 2024-05-04T12:15:17.027 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:16 smithi146 ceph-mon[35311]: mds.? [v2:172.21.15.146:6824/4162809119,v1:172.21.15.146:6825/4162809119] up:standby 2024-05-04T12:15:17.027 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:16 smithi146 ceph-mon[35311]: Dropping low affinity active daemon mds.cephfs.smithi079.pspich in favor of higher affinity standby. 2024-05-04T12:15:17.027 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:16 smithi146 ceph-mon[35311]: Replacing daemon mds.cephfs.smithi079.pspich as rank 0 with standby daemon mds.cephfs.smithi079.hzcmip 2024-05-04T12:15:17.027 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:16 smithi146 ceph-mon[35311]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-05-04T12:15:17.027 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:16 smithi146 ceph-mon[35311]: fsmap cephfs:1 {0=cephfs.smithi079.pspich=up:active} 3 up:standby 2024-05-04T12:15:17.027 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:16 smithi146 ceph-mon[35311]: pgmap v186: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.4 KiB/s wr, 4 op/s 2024-05-04T12:15:17.027 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:16 smithi146 ceph-mon[35311]: osdmap e46: 6 total, 6 up, 6 in 2024-05-04T12:15:17.027 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:16 smithi146 ceph-mon[35311]: fsmap cephfs:1/1 {0=cephfs.smithi079.hzcmip=up:replay} 2 up:standby 2024-05-04T12:15:17.027 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:16 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:15:17.028 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:16 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:15:17.169 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:17.170+0000 7f92c1d9b700 1 -- 172.21.15.79:0/1098741230 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f92bc0ff5a0 msgr2=0x7f92bc0ff9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:15:17.169 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:17.170+0000 7f92c1d9b700 1 --2- 172.21.15.79:0/1098741230 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f92bc0ff5a0 0x7f92bc0ff9c0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f92b0009910 tx=0x7f92b0009c20 comp rx=0 tx=0).stop 2024-05-04T12:15:17.169 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:17.170+0000 7f92c1d9b700 1 -- 172.21.15.79:0/1098741230 shutdown_connections 2024-05-04T12:15:17.169 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:17.170+0000 7f92c1d9b700 1 --2- 172.21.15.79:0/1098741230 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f92bc0fff00 0x7f92bc100380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:17.170 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:17.170+0000 7f92c1d9b700 1 --2- 172.21.15.79:0/1098741230 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f92bc0ff5a0 0x7f92bc0ff9c0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:17.170 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:17.170+0000 7f92c1d9b700 1 -- 172.21.15.79:0/1098741230 >> 172.21.15.79:0/1098741230 conn(0x7f92bc0fb120 msgr2=0x7f92bc0fd580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:15:17.170 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:17.170+0000 7f92c1d9b700 1 -- 172.21.15.79:0/1098741230 shutdown_connections 2024-05-04T12:15:17.170 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:17.170+0000 7f92c1d9b700 1 -- 172.21.15.79:0/1098741230 wait complete. 2024-05-04T12:15:17.172 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:17.170+0000 7f92c1d9b700 1 Processor -- start 2024-05-04T12:15:17.172 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:17.171+0000 7f92c1d9b700 1 -- start start 2024-05-04T12:15:17.172 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:17.171+0000 7f92c1d9b700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f92bc0ff5a0 0x7f92bc1986a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:15:17.172 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:17.171+0000 7f92c1d9b700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f92bc0fff00 0x7f92bc198be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:15:17.172 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:17.171+0000 7f92c1d9b700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92bc199200 con 0x7f92bc0fff00 2024-05-04T12:15:17.172 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:17.171+0000 7f92c1d9b700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92bc199340 con 0x7f92bc0ff5a0 2024-05-04T12:15:17.172 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:17.171+0000 7f92baffd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f92bc0fff00 0x7f92bc198be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:15:17.172 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:17.171+0000 7f92baffd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f92bc0fff00 0x7f92bc198be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:35796/0 (socket says 172.21.15.79:35796) 2024-05-04T12:15:17.172 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:17.171+0000 7f92baffd700 1 -- 172.21.15.79:0/3310014061 learned_addr learned my addr 172.21.15.79:0/3310014061 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:15:17.172 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:17.171+0000 7f92bb7fe700 1 --2- 172.21.15.79:0/3310014061 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f92bc0ff5a0 0x7f92bc1986a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:15:17.172 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:17.171+0000 7f92baffd700 1 -- 172.21.15.79:0/3310014061 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f92bc0ff5a0 msgr2=0x7f92bc1986a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:15:17.172 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:17.171+0000 7f92baffd700 1 --2- 172.21.15.79:0/3310014061 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f92bc0ff5a0 0x7f92bc1986a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:17.172 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:17.171+0000 7f92baffd700 1 -- 172.21.15.79:0/3310014061 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92b00095f0 con 0x7f92bc0fff00 2024-05-04T12:15:17.173 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:17.172+0000 7f92baffd700 1 --2- 172.21.15.79:0/3310014061 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f92bc0fff00 0x7f92bc198be0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f92ac00d8d0 tx=0x7f92ac00dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:15:17.173 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:17.172+0000 7f92b8ff9700 1 -- 172.21.15.79:0/3310014061 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f92ac0098a0 con 0x7f92bc0fff00 2024-05-04T12:15:17.173 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:17.172+0000 7f92c1d9b700 1 -- 172.21.15.79:0/3310014061 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92bc19ddf0 con 0x7f92bc0fff00 2024-05-04T12:15:17.173 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:17.172+0000 7f92b8ff9700 1 -- 172.21.15.79:0/3310014061 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f92ac010460 con 0x7f92bc0fff00 2024-05-04T12:15:17.173 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:17.172+0000 7f92b8ff9700 1 -- 172.21.15.79:0/3310014061 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f92ac00f5f0 con 0x7f92bc0fff00 2024-05-04T12:15:17.173 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:17.172+0000 7f92c1d9b700 1 -- 172.21.15.79:0/3310014061 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92bc19e310 con 0x7f92bc0fff00 2024-05-04T12:15:17.174 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:17.173+0000 7f92c1d9b700 1 -- 172.21.15.79:0/3310014061 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92bc10ba00 con 0x7f92bc0fff00 2024-05-04T12:15:17.174 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:17.173+0000 7f92b8ff9700 1 -- 172.21.15.79:0/3310014061 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f92ac00f820 con 0x7f92bc0fff00 2024-05-04T12:15:17.174 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:17.174+0000 7f92b8ff9700 1 --2- 172.21.15.79:0/3310014061 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f92a406c4e0 0x7f92a406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:15:17.174 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:17.174+0000 7f92b8ff9700 1 -- 172.21.15.79:0/3310014061 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f92ac08b770 con 0x7f92bc0fff00 2024-05-04T12:15:17.174 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:17.174+0000 7f92bb7fe700 1 --2- 172.21.15.79:0/3310014061 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f92a406c4e0 0x7f92a406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:15:17.174 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:17.174+0000 7f92bb7fe700 1 --2- 172.21.15.79:0/3310014061 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f92a406c4e0 0x7f92a406e9a0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f92b0006010 tx=0x7f92b000b540 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:15:17.181 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:17.182+0000 7f92b8ff9700 1 -- 172.21.15.79:0/3310014061 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f92ac059980 con 0x7f92bc0fff00 2024-05-04T12:15:17.428 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:17.429+0000 7f92c1d9b700 1 -- 172.21.15.79:0/3310014061 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f92bc19e8f0 con 0x7f92bc0fff00 2024-05-04T12:15:17.429 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:17.429+0000 7f92b8ff9700 1 -- 172.21.15.79:0/3310014061 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 12 v12) v1 ==== 76+0+1814 (secure 0 0 0) 0x7f92ac059510 con 0x7f92bc0fff00 2024-05-04T12:15:17.429 INFO:teuthology.orchestra.run.smithi079.stdout:e12 2024-05-04T12:15:17.429 INFO:teuthology.orchestra.run.smithi079.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-05-04T12:15:17.429 INFO:teuthology.orchestra.run.smithi079.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-05-04T12:15:17.429 INFO:teuthology.orchestra.run.smithi079.stdout:legacy client fscid: 1 2024-05-04T12:15:17.430 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-05-04T12:15:17.430 INFO:teuthology.orchestra.run.smithi079.stdout:Filesystem 'cephfs' (1) 2024-05-04T12:15:17.430 INFO:teuthology.orchestra.run.smithi079.stdout:fs_name cephfs 2024-05-04T12:15:17.430 INFO:teuthology.orchestra.run.smithi079.stdout:epoch 12 2024-05-04T12:15:17.430 INFO:teuthology.orchestra.run.smithi079.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-05-04T12:15:17.430 INFO:teuthology.orchestra.run.smithi079.stdout:created 2024-05-04T12:15:00.083118+0000 2024-05-04T12:15:17.430 INFO:teuthology.orchestra.run.smithi079.stdout:modified 2024-05-04T12:15:16.642824+0000 2024-05-04T12:15:17.430 INFO:teuthology.orchestra.run.smithi079.stdout:tableserver 0 2024-05-04T12:15:17.430 INFO:teuthology.orchestra.run.smithi079.stdout:root 0 2024-05-04T12:15:17.430 INFO:teuthology.orchestra.run.smithi079.stdout:session_timeout 60 2024-05-04T12:15:17.430 INFO:teuthology.orchestra.run.smithi079.stdout:session_autoclose 300 2024-05-04T12:15:17.430 INFO:teuthology.orchestra.run.smithi079.stdout:max_file_size 1099511627776 2024-05-04T12:15:17.430 INFO:teuthology.orchestra.run.smithi079.stdout:required_client_features {} 2024-05-04T12:15:17.430 INFO:teuthology.orchestra.run.smithi079.stdout:last_failure 0 2024-05-04T12:15:17.431 INFO:teuthology.orchestra.run.smithi079.stdout:last_failure_osd_epoch 46 2024-05-04T12:15:17.431 INFO:teuthology.orchestra.run.smithi079.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-05-04T12:15:17.431 INFO:teuthology.orchestra.run.smithi079.stdout:max_mds 1 2024-05-04T12:15:17.431 INFO:teuthology.orchestra.run.smithi079.stdout:in 0 2024-05-04T12:15:17.431 INFO:teuthology.orchestra.run.smithi079.stdout:up {0=14482} 2024-05-04T12:15:17.431 INFO:teuthology.orchestra.run.smithi079.stdout:failed 2024-05-04T12:15:17.431 INFO:teuthology.orchestra.run.smithi079.stdout:damaged 2024-05-04T12:15:17.431 INFO:teuthology.orchestra.run.smithi079.stdout:stopped 2024-05-04T12:15:17.431 INFO:teuthology.orchestra.run.smithi079.stdout:data_pools [3] 2024-05-04T12:15:17.431 INFO:teuthology.orchestra.run.smithi079.stdout:metadata_pool 2 2024-05-04T12:15:17.431 INFO:teuthology.orchestra.run.smithi079.stdout:inline_data enabled 2024-05-04T12:15:17.431 INFO:teuthology.orchestra.run.smithi079.stdout:balancer 2024-05-04T12:15:17.431 INFO:teuthology.orchestra.run.smithi079.stdout:bal_rank_mask -1 2024-05-04T12:15:17.431 INFO:teuthology.orchestra.run.smithi079.stdout:standby_count_wanted 1 2024-05-04T12:15:17.432 INFO:teuthology.orchestra.run.smithi079.stdout:[mds.cephfs.smithi079.hzcmip{0:14482} state up:reconnect seq 3 join_fscid=1 addr [v2:172.21.15.79:6828/4287465339,v1:172.21.15.79:6829/4287465339] compat {c=[1],r=[1],i=[7ff]}] 2024-05-04T12:15:17.432 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-05-04T12:15:17.432 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-05-04T12:15:17.432 INFO:teuthology.orchestra.run.smithi079.stdout:Standby daemons: 2024-05-04T12:15:17.432 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-05-04T12:15:17.432 INFO:teuthology.orchestra.run.smithi079.stdout:[mds.cephfs.smithi079.pspich{-1:14492} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.79:6826/732688410,v1:172.21.15.79:6827/732688410] compat {c=[1],r=[1],i=[7ff]}] 2024-05-04T12:15:17.432 INFO:teuthology.orchestra.run.smithi079.stdout:[mds.cephfs.smithi146.tfajce{-1:24271} state up:standby seq 3 join_fscid=1 addr [v2:172.21.15.146:6824/4162809119,v1:172.21.15.146:6825/4162809119] compat {c=[1],r=[1],i=[7ff]}] 2024-05-04T12:15:17.432 INFO:teuthology.orchestra.run.smithi079.stdout:[mds.cephfs.smithi146.mkcmzm{-1:24283} state up:standby seq 1 addr [v2:172.21.15.146:6826/178032877,v1:172.21.15.146:6827/178032877] compat {c=[1],r=[1],i=[7ff]}] 2024-05-04T12:15:17.433 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:17.432+0000 7f92c1d9b700 1 -- 172.21.15.79:0/3310014061 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f92a406c4e0 msgr2=0x7f92a406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:15:17.433 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:17.432+0000 7f92c1d9b700 1 --2- 172.21.15.79:0/3310014061 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f92a406c4e0 0x7f92a406e9a0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f92b0006010 tx=0x7f92b000b540 comp rx=0 tx=0).stop 2024-05-04T12:15:17.433 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:17.432+0000 7f92c1d9b700 1 -- 172.21.15.79:0/3310014061 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f92bc0fff00 msgr2=0x7f92bc198be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:15:17.433 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:17.432+0000 7f92c1d9b700 1 --2- 172.21.15.79:0/3310014061 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f92bc0fff00 0x7f92bc198be0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f92ac00d8d0 tx=0x7f92ac00dbe0 comp rx=0 tx=0).stop 2024-05-04T12:15:17.433 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:17.432+0000 7f92c1d9b700 1 -- 172.21.15.79:0/3310014061 shutdown_connections 2024-05-04T12:15:17.433 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:17.432+0000 7f92c1d9b700 1 --2- 172.21.15.79:0/3310014061 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f92a406c4e0 0x7f92a406e9a0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:17.433 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:17.432+0000 7f92c1d9b700 1 --2- 172.21.15.79:0/3310014061 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f92bc0ff5a0 0x7f92bc1986a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:17.433 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:17.432+0000 7f92c1d9b700 1 --2- 172.21.15.79:0/3310014061 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f92bc0fff00 0x7f92bc198be0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:17.433 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:17.432+0000 7f92c1d9b700 1 -- 172.21.15.79:0/3310014061 >> 172.21.15.79:0/3310014061 conn(0x7f92bc0fb120 msgr2=0x7f92bc107bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:15:17.433 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:17.433+0000 7f92c1d9b700 1 -- 172.21.15.79:0/3310014061 shutdown_connections 2024-05-04T12:15:17.433 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:17.433+0000 7f92c1d9b700 1 -- 172.21.15.79:0/3310014061 wait complete. 2024-05-04T12:15:17.434 INFO:teuthology.orchestra.run.smithi079.stderr:dumped fsmap epoch 12 2024-05-04T12:15:17.695 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:17 smithi079 ceph-mon[27222]: mds.? [v2:172.21.15.79:6828/4287465339,v1:172.21.15.79:6829/4287465339] up:reconnect 2024-05-04T12:15:17.695 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:17 smithi079 ceph-mon[27222]: mds.? [v2:172.21.15.79:6826/732688410,v1:172.21.15.79:6827/732688410] up:boot 2024-05-04T12:15:17.695 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:17 smithi079 ceph-mon[27222]: fsmap cephfs:1/1 {0=cephfs.smithi079.hzcmip=up:reconnect} 3 up:standby 2024-05-04T12:15:17.696 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:17 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi079.pspich"}]: dispatch 2024-05-04T12:15:17.696 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:17 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:15:17.696 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:17 smithi079 ceph-mon[27222]: from='client.? 172.21.15.79:0/3310014061' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-05-04T12:15:18.027 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:17 smithi146 ceph-mon[35311]: mds.? [v2:172.21.15.79:6828/4287465339,v1:172.21.15.79:6829/4287465339] up:reconnect 2024-05-04T12:15:18.027 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:17 smithi146 ceph-mon[35311]: mds.? [v2:172.21.15.79:6826/732688410,v1:172.21.15.79:6827/732688410] up:boot 2024-05-04T12:15:18.027 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:17 smithi146 ceph-mon[35311]: fsmap cephfs:1/1 {0=cephfs.smithi079.hzcmip=up:reconnect} 3 up:standby 2024-05-04T12:15:18.027 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:17 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi079.pspich"}]: dispatch 2024-05-04T12:15:18.027 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:17 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:15:18.027 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:17 smithi146 ceph-mon[35311]: from='client.? 172.21.15.79:0/3310014061' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-05-04T12:15:18.387 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d9b985c4-0a0e-11ef-bc97-c7b262605968 -- bash -c 'ceph --format=json fs dump | jq -e ".filesystems | length == 1"' 2024-05-04T12:15:18.945 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:18 smithi079 ceph-mon[27222]: pgmap v187: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1023 B/s rd, 818 B/s wr, 1 op/s 2024-05-04T12:15:18.945 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:18 smithi079 ceph-mon[27222]: mds.? [v2:172.21.15.79:6828/4287465339,v1:172.21.15.79:6829/4287465339] up:rejoin 2024-05-04T12:15:18.945 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:18 smithi079 ceph-mon[27222]: mds.? [v2:172.21.15.146:6826/178032877,v1:172.21.15.146:6827/178032877] up:standby 2024-05-04T12:15:18.945 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:18 smithi079 ceph-mon[27222]: fsmap cephfs:1/1 {0=cephfs.smithi079.hzcmip=up:rejoin} 3 up:standby 2024-05-04T12:15:18.945 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:18 smithi079 ceph-mon[27222]: daemon mds.cephfs.smithi079.hzcmip is now active in filesystem cephfs as rank 0 2024-05-04T12:15:19.027 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:18 smithi146 ceph-mon[35311]: pgmap v187: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1023 B/s rd, 818 B/s wr, 1 op/s 2024-05-04T12:15:19.027 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:18 smithi146 ceph-mon[35311]: mds.? [v2:172.21.15.79:6828/4287465339,v1:172.21.15.79:6829/4287465339] up:rejoin 2024-05-04T12:15:19.027 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:18 smithi146 ceph-mon[35311]: mds.? [v2:172.21.15.146:6826/178032877,v1:172.21.15.146:6827/178032877] up:standby 2024-05-04T12:15:19.027 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:18 smithi146 ceph-mon[35311]: fsmap cephfs:1/1 {0=cephfs.smithi079.hzcmip=up:rejoin} 3 up:standby 2024-05-04T12:15:19.027 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:18 smithi146 ceph-mon[35311]: daemon mds.cephfs.smithi079.hzcmip is now active in filesystem cephfs as rank 0 2024-05-04T12:15:19.792 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/d9b985c4-0a0e-11ef-bc97-c7b262605968/mon.smithi079/config 2024-05-04T12:15:19.916 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:19 smithi079 ceph-mon[27222]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-05-04T12:15:19.916 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:19 smithi079 ceph-mon[27222]: mds.? [v2:172.21.15.79:6828/4287465339,v1:172.21.15.79:6829/4287465339] up:active 2024-05-04T12:15:19.917 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:19 smithi079 ceph-mon[27222]: fsmap cephfs:1 {0=cephfs.smithi079.hzcmip=up:active} 3 up:standby 2024-05-04T12:15:20.027 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:19 smithi146 ceph-mon[35311]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-05-04T12:15:20.027 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:19 smithi146 ceph-mon[35311]: mds.? [v2:172.21.15.79:6828/4287465339,v1:172.21.15.79:6829/4287465339] up:active 2024-05-04T12:15:20.027 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:19 smithi146 ceph-mon[35311]: fsmap cephfs:1 {0=cephfs.smithi079.hzcmip=up:active} 3 up:standby 2024-05-04T12:15:20.945 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:20 smithi079 ceph-mon[27222]: pgmap v188: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-05-04T12:15:21.027 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:20 smithi146 ceph-mon[35311]: pgmap v188: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-05-04T12:15:21.858 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:21.855+0000 7f6e3728f700 1 -- 172.21.15.79:0/1772418309 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6e30104070 msgr2=0x7f6e30106460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:15:21.858 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:21.855+0000 7f6e3728f700 1 --2- 172.21.15.79:0/1772418309 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6e30104070 0x7f6e30106460 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f6e20009910 tx=0x7f6e20009c20 comp rx=0 tx=0).stop 2024-05-04T12:15:21.859 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:21.855+0000 7f6e3728f700 1 -- 172.21.15.79:0/1772418309 shutdown_connections 2024-05-04T12:15:21.859 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:21.855+0000 7f6e3728f700 1 --2- 172.21.15.79:0/1772418309 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6e30104070 0x7f6e30106460 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:21.859 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:21.855+0000 7f6e3728f700 1 --2- 172.21.15.79:0/1772418309 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6e30101740 0x7f6e30103b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:21.859 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:21.855+0000 7f6e3728f700 1 -- 172.21.15.79:0/1772418309 >> 172.21.15.79:0/1772418309 conn(0x7f6e300fb150 msgr2=0x7f6e300fd590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:15:21.859 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:21.855+0000 7f6e3728f700 1 -- 172.21.15.79:0/1772418309 shutdown_connections 2024-05-04T12:15:21.859 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:21.855+0000 7f6e3728f700 1 -- 172.21.15.79:0/1772418309 wait complete. 2024-05-04T12:15:21.860 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:21.855+0000 7f6e3728f700 1 Processor -- start 2024-05-04T12:15:21.860 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:21.855+0000 7f6e3728f700 1 -- start start 2024-05-04T12:15:21.860 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:21.856+0000 7f6e3728f700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6e30101740 0x7f6e30196460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:15:21.860 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:21.856+0000 7f6e3728f700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6e30104070 0x7f6e301969a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:15:21.860 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:21.856+0000 7f6e3728f700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e30196fc0 con 0x7f6e30101740 2024-05-04T12:15:21.860 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:21.856+0000 7f6e3728f700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e30197100 con 0x7f6e30104070 2024-05-04T12:15:21.861 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:21.856+0000 7f6e3502b700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6e30101740 0x7f6e30196460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:15:21.861 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:21.856+0000 7f6e3502b700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6e30101740 0x7f6e30196460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:35806/0 (socket says 172.21.15.79:35806) 2024-05-04T12:15:21.861 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:21.856+0000 7f6e3502b700 1 -- 172.21.15.79:0/4037844233 learned_addr learned my addr 172.21.15.79:0/4037844233 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:15:21.861 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:21.856+0000 7f6e3482a700 1 --2- 172.21.15.79:0/4037844233 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6e30104070 0x7f6e301969a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:15:21.861 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:21.856+0000 7f6e3502b700 1 -- 172.21.15.79:0/4037844233 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6e30104070 msgr2=0x7f6e301969a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:15:21.861 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:21.856+0000 7f6e3502b700 1 --2- 172.21.15.79:0/4037844233 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6e30104070 0x7f6e301969a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:21.861 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:21.856+0000 7f6e3502b700 1 -- 172.21.15.79:0/4037844233 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e200095f0 con 0x7f6e30101740 2024-05-04T12:15:21.862 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:21.856+0000 7f6e3482a700 1 --2- 172.21.15.79:0/4037844233 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6e30104070 0x7f6e301969a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-04T12:15:21.862 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:21.856+0000 7f6e3502b700 1 --2- 172.21.15.79:0/4037844233 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6e30101740 0x7f6e30196460 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f6e2c00c8a0 tx=0x7f6e2c00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:15:21.862 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:21.856+0000 7f6e267fc700 1 -- 172.21.15.79:0/4037844233 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6e2c00cea0 con 0x7f6e30101740 2024-05-04T12:15:21.862 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:21.856+0000 7f6e3728f700 1 -- 172.21.15.79:0/4037844233 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e3019bbb0 con 0x7f6e30101740 2024-05-04T12:15:21.862 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:21.856+0000 7f6e267fc700 1 -- 172.21.15.79:0/4037844233 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f6e2c004830 con 0x7f6e30101740 2024-05-04T12:15:21.862 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:21.856+0000 7f6e267fc700 1 -- 172.21.15.79:0/4037844233 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6e2c0055b0 con 0x7f6e30101740 2024-05-04T12:15:21.862 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:21.857+0000 7f6e3728f700 1 -- 172.21.15.79:0/4037844233 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e3019c100 con 0x7f6e30101740 2024-05-04T12:15:21.863 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:21.858+0000 7f6e3728f700 1 -- 172.21.15.79:0/4037844233 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e30190620 con 0x7f6e30101740 2024-05-04T12:15:21.863 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:21.858+0000 7f6e267fc700 1 -- 172.21.15.79:0/4037844233 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f6e2c007ad0 con 0x7f6e30101740 2024-05-04T12:15:21.863 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:21.858+0000 7f6e267fc700 1 --2- 172.21.15.79:0/4037844233 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f6e1c06c4e0 0x7f6e1c06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:15:21.863 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:21.858+0000 7f6e267fc700 1 -- 172.21.15.79:0/4037844233 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f6e2c08b600 con 0x7f6e30101740 2024-05-04T12:15:21.863 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:21.858+0000 7f6e3482a700 1 --2- 172.21.15.79:0/4037844233 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f6e1c06c4e0 0x7f6e1c06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:15:21.863 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:21.859+0000 7f6e3482a700 1 --2- 172.21.15.79:0/4037844233 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f6e1c06c4e0 0x7f6e1c06e9a0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f6e2000b5c0 tx=0x7f6e20005e20 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:15:21.864 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:21.863+0000 7f6e267fc700 1 -- 172.21.15.79:0/4037844233 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f6e2c059810 con 0x7f6e30101740 2024-05-04T12:15:22.111 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:22.111+0000 7f6e3728f700 1 -- 172.21.15.79:0/4037844233 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6e3002cc70 con 0x7f6e30101740 2024-05-04T12:15:22.111 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:22.112+0000 7f6e267fc700 1 -- 172.21.15.79:0/4037844233 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) v1 ==== 94+0+4739 (secure 0 0 0) 0x7f6e2c0593a0 con 0x7f6e30101740 2024-05-04T12:15:22.114 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:22.114+0000 7f6e3728f700 1 -- 172.21.15.79:0/4037844233 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f6e1c06c4e0 msgr2=0x7f6e1c06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:15:22.114 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:22.114+0000 7f6e3728f700 1 --2- 172.21.15.79:0/4037844233 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f6e1c06c4e0 0x7f6e1c06e9a0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f6e2000b5c0 tx=0x7f6e20005e20 comp rx=0 tx=0).stop 2024-05-04T12:15:22.114 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:22.114+0000 7f6e3728f700 1 -- 172.21.15.79:0/4037844233 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6e30101740 msgr2=0x7f6e30196460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:15:22.114 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:22.114+0000 7f6e3728f700 1 --2- 172.21.15.79:0/4037844233 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6e30101740 0x7f6e30196460 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f6e2c00c8a0 tx=0x7f6e2c00cc60 comp rx=0 tx=0).stop 2024-05-04T12:15:22.114 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:22.114+0000 7f6e3728f700 1 -- 172.21.15.79:0/4037844233 shutdown_connections 2024-05-04T12:15:22.114 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:22.114+0000 7f6e3728f700 1 --2- 172.21.15.79:0/4037844233 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f6e1c06c4e0 0x7f6e1c06e9a0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:22.114 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:22.114+0000 7f6e3728f700 1 --2- 172.21.15.79:0/4037844233 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6e30101740 0x7f6e30196460 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:22.115 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:22.114+0000 7f6e3728f700 1 --2- 172.21.15.79:0/4037844233 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6e30104070 0x7f6e301969a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:22.115 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:22.114+0000 7f6e3728f700 1 -- 172.21.15.79:0/4037844233 >> 172.21.15.79:0/4037844233 conn(0x7f6e300fb150 msgr2=0x7f6e300fd590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:15:22.115 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:22.114+0000 7f6e3728f700 1 -- 172.21.15.79:0/4037844233 shutdown_connections 2024-05-04T12:15:22.115 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:22.114+0000 7f6e3728f700 1 -- 172.21.15.79:0/4037844233 wait complete. 2024-05-04T12:15:22.115 INFO:teuthology.orchestra.run.smithi079.stderr:dumped fsmap epoch 14 2024-05-04T12:15:22.125 INFO:teuthology.orchestra.run.smithi079.stdout:true 2024-05-04T12:15:22.744 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d9b985c4-0a0e-11ef-bc97-c7b262605968 -- bash -c 'while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done' 2024-05-04T12:15:22.974 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:22 smithi146 ceph-mon[35311]: pgmap v189: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 4.6 KiB/s rd, 0 B/s wr, 4 op/s 2024-05-04T12:15:22.975 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:22 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:15:22.975 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:22 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:15:22.975 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:22 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-04T12:15:22.975 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:22 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-04T12:15:22.975 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:22 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:15:22.975 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:22 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-04T12:15:22.975 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:22 smithi146 ceph-mon[35311]: from='client.? 172.21.15.79:0/4037844233' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-04T12:15:22.975 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/d9b985c4-0a0e-11ef-bc97-c7b262605968/mon.smithi079/config 2024-05-04T12:15:23.325 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:22 smithi079 ceph-mon[27222]: pgmap v189: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 4.6 KiB/s rd, 0 B/s wr, 4 op/s 2024-05-04T12:15:23.325 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:22 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:15:23.325 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:22 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:15:23.325 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:22 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-04T12:15:23.325 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:22 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-04T12:15:23.325 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:22 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:15:23.325 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:22 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-04T12:15:23.326 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:22 smithi079 ceph-mon[27222]: from='client.? 172.21.15.79:0/4037844233' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-04T12:15:25.235 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:24 smithi079 ceph-mon[27222]: pgmap v190: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 8.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-05-04T12:15:25.235 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:24 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:15:25.236 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:24 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:15:25.276 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:24 smithi146 ceph-mon[35311]: pgmap v190: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 8.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-05-04T12:15:25.277 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:24 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:15:25.277 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:24 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:15:26.151 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:26.150+0000 7efd11b1e700 1 -- 172.21.15.79:0/424476317 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efd0c100e10 msgr2=0x7efd0c101270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:15:26.151 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:26.150+0000 7efd11b1e700 1 --2- 172.21.15.79:0/424476317 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efd0c100e10 0x7efd0c101270 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7efcfc009910 tx=0x7efcfc009c20 comp rx=0 tx=0).stop 2024-05-04T12:15:26.151 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:26.150+0000 7efd11b1e700 1 -- 172.21.15.79:0/424476317 shutdown_connections 2024-05-04T12:15:26.151 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:26.150+0000 7efd11b1e700 1 --2- 172.21.15.79:0/424476317 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efd0c100e10 0x7efd0c101270 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:26.151 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:26.150+0000 7efd11b1e700 1 --2- 172.21.15.79:0/424476317 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efd0c0ffc10 0x7efd0c100030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:26.151 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:26.150+0000 7efd11b1e700 1 -- 172.21.15.79:0/424476317 >> 172.21.15.79:0/424476317 conn(0x7efd0c0fb190 msgr2=0x7efd0c0fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:15:26.152 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:26.150+0000 7efd11b1e700 1 -- 172.21.15.79:0/424476317 shutdown_connections 2024-05-04T12:15:26.152 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:26.150+0000 7efd11b1e700 1 -- 172.21.15.79:0/424476317 wait complete. 2024-05-04T12:15:26.152 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:26.150+0000 7efd11b1e700 1 Processor -- start 2024-05-04T12:15:26.152 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:26.151+0000 7efd11b1e700 1 -- start start 2024-05-04T12:15:26.152 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:26.151+0000 7efd11b1e700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efd0c0ffc10 0x7efd0c0690e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:15:26.152 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:26.151+0000 7efd11b1e700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efd0c100e10 0x7efd0c19b200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:15:26.152 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:26.151+0000 7efd11b1e700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd0c19b7d0 con 0x7efd0c100e10 2024-05-04T12:15:26.152 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:26.151+0000 7efd11b1e700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd0c19b910 con 0x7efd0c0ffc10 2024-05-04T12:15:26.152 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:26.151+0000 7efd0affd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efd0c100e10 0x7efd0c19b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:15:26.152 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:26.151+0000 7efd0affd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efd0c100e10 0x7efd0c19b200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:51338/0 (socket says 172.21.15.79:51338) 2024-05-04T12:15:26.152 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:26.151+0000 7efd0affd700 1 -- 172.21.15.79:0/1058536457 learned_addr learned my addr 172.21.15.79:0/1058536457 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:15:26.152 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:26.151+0000 7efd0b7fe700 1 --2- 172.21.15.79:0/1058536457 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efd0c0ffc10 0x7efd0c0690e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:15:26.152 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:26.151+0000 7efd0affd700 1 -- 172.21.15.79:0/1058536457 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efd0c0ffc10 msgr2=0x7efd0c0690e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:15:26.153 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:26.151+0000 7efd0affd700 1 --2- 172.21.15.79:0/1058536457 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efd0c0ffc10 0x7efd0c0690e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:26.153 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:26.151+0000 7efd0affd700 1 -- 172.21.15.79:0/1058536457 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcfc0095f0 con 0x7efd0c100e10 2024-05-04T12:15:26.153 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:26.151+0000 7efd0affd700 1 --2- 172.21.15.79:0/1058536457 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efd0c100e10 0x7efd0c19b200 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7efcfc005b40 tx=0x7efcfc00bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:15:26.153 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:26.152+0000 7efd08ff9700 1 -- 172.21.15.79:0/1058536457 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efcfc01d070 con 0x7efd0c100e10 2024-05-04T12:15:26.153 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:26.152+0000 7efd08ff9700 1 -- 172.21.15.79:0/1058536457 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7efcfc00f460 con 0x7efd0c100e10 2024-05-04T12:15:26.153 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:26.152+0000 7efd08ff9700 1 -- 172.21.15.79:0/1058536457 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efcfc021600 con 0x7efd0c100e10 2024-05-04T12:15:26.153 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:26.152+0000 7efd11b1e700 1 -- 172.21.15.79:0/1058536457 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd0c19bb90 con 0x7efd0c100e10 2024-05-04T12:15:26.154 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:26.152+0000 7efd11b1e700 1 -- 172.21.15.79:0/1058536457 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd0c19c080 con 0x7efd0c100e10 2024-05-04T12:15:26.154 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:26.153+0000 7efd11b1e700 1 -- 172.21.15.79:0/1058536457 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd0c108700 con 0x7efd0c100e10 2024-05-04T12:15:26.154 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:26.153+0000 7efd08ff9700 1 -- 172.21.15.79:0/1058536457 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7efcfc00f5d0 con 0x7efd0c100e10 2024-05-04T12:15:26.154 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:26.154+0000 7efd08ff9700 1 --2- 172.21.15.79:0/1058536457 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7efcf406c4e0 0x7efcf406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:15:26.154 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:26.154+0000 7efd08ff9700 1 -- 172.21.15.79:0/1058536457 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7efcfc08ca70 con 0x7efd0c100e10 2024-05-04T12:15:26.154 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:26.154+0000 7efd0b7fe700 1 --2- 172.21.15.79:0/1058536457 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7efcf406c4e0 0x7efcf406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:15:26.155 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:26.154+0000 7efd0b7fe700 1 --2- 172.21.15.79:0/1058536457 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7efcf406c4e0 0x7efcf406e9a0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7efd0c100c70 tx=0x7efd0000b410 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:15:26.157 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:26.158+0000 7efd08ff9700 1 -- 172.21.15.79:0/1058536457 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7efcfc05ad00 con 0x7efd0c100e10 2024-05-04T12:15:26.430 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:26.431+0000 7efd11b1e700 1 -- 172.21.15.79:0/1058536457 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "mds versions", "format": "json"} v 0) v1 -- 0x7efd0c19c6e0 con 0x7efd0c100e10 2024-05-04T12:15:26.430 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:26.431+0000 7efd08ff9700 1 -- 172.21.15.79:0/1058536457 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "mds versions", "format": "json"}]=0 v14) v1 ==== 78+0+83 (secure 0 0 0) 0x7efcfc00f880 con 0x7efd0c100e10 2024-05-04T12:15:26.432 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:26.433+0000 7efd11b1e700 1 -- 172.21.15.79:0/1058536457 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7efcf406c4e0 msgr2=0x7efcf406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:15:26.433 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:26.433+0000 7efd11b1e700 1 --2- 172.21.15.79:0/1058536457 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7efcf406c4e0 0x7efcf406e9a0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7efd0c100c70 tx=0x7efd0000b410 comp rx=0 tx=0).stop 2024-05-04T12:15:26.433 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:26.433+0000 7efd11b1e700 1 -- 172.21.15.79:0/1058536457 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efd0c100e10 msgr2=0x7efd0c19b200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:15:26.433 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:26.433+0000 7efd11b1e700 1 --2- 172.21.15.79:0/1058536457 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efd0c100e10 0x7efd0c19b200 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7efcfc005b40 tx=0x7efcfc00bd70 comp rx=0 tx=0).stop 2024-05-04T12:15:26.433 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:26.433+0000 7efd11b1e700 1 -- 172.21.15.79:0/1058536457 shutdown_connections 2024-05-04T12:15:26.433 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:26.433+0000 7efd11b1e700 1 --2- 172.21.15.79:0/1058536457 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7efcf406c4e0 0x7efcf406e9a0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:26.433 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:26.433+0000 7efd11b1e700 1 --2- 172.21.15.79:0/1058536457 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efd0c0ffc10 0x7efd0c0690e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:26.433 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:26.433+0000 7efd11b1e700 1 --2- 172.21.15.79:0/1058536457 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efd0c100e10 0x7efd0c19b200 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:26.433 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:26.433+0000 7efd11b1e700 1 -- 172.21.15.79:0/1058536457 >> 172.21.15.79:0/1058536457 conn(0x7efd0c0fb190 msgr2=0x7efd0c104040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:15:26.433 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:26.433+0000 7efd11b1e700 1 -- 172.21.15.79:0/1058536457 shutdown_connections 2024-05-04T12:15:26.433 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:26.433+0000 7efd11b1e700 1 -- 172.21.15.79:0/1058536457 wait complete. 2024-05-04T12:15:26.443 INFO:teuthology.orchestra.run.smithi079.stdout:true 2024-05-04T12:15:27.066 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:27 smithi079 ceph-mon[27222]: pgmap v191: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 8.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-05-04T12:15:27.067 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:27 smithi079 ceph-mon[27222]: from='client.? 172.21.15.79:0/1058536457' entity='client.admin' cmd=[{"prefix": "mds versions", "format": "json"}]: dispatch 2024-05-04T12:15:27.067 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:27 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:15:27.067 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:27 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-04T12:15:27.085 INFO:teuthology.run_tasks:Running task fs.pre_upgrade_save... 2024-05-04T12:15:27.097 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-04T12:15:27.214 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:27.213+0000 7f52f2992640 1 -- 172.21.15.79:0/3101540771 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f52ec1012a0 msgr2=0x7f52ec103710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:15:27.214 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:27.213+0000 7f52f2992640 1 --2- 172.21.15.79:0/3101540771 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f52ec1012a0 0x7f52ec103710 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f52e0009880 tx=0x7f52e002bce0 comp rx=0 tx=0).stop 2024-05-04T12:15:27.214 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:27.213+0000 7f52f2992640 1 -- 172.21.15.79:0/3101540771 shutdown_connections 2024-05-04T12:15:27.214 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:27.213+0000 7f52f2992640 1 --2- 172.21.15.79:0/3101540771 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f52ec1012a0 0x7f52ec103710 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:27.214 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:27.213+0000 7f52f2992640 1 --2- 172.21.15.79:0/3101540771 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f52ec0fe930 0x7f52ec100d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:27.214 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:27.213+0000 7f52f2992640 1 -- 172.21.15.79:0/3101540771 >> 172.21.15.79:0/3101540771 conn(0x7f52ec0fa530 msgr2=0x7f52ec0fc970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:15:27.214 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:27.213+0000 7f52f2992640 1 -- 172.21.15.79:0/3101540771 shutdown_connections 2024-05-04T12:15:27.214 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:27.213+0000 7f52f2992640 1 -- 172.21.15.79:0/3101540771 wait complete. 2024-05-04T12:15:27.214 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:27.213+0000 7f52f2992640 1 Processor -- start 2024-05-04T12:15:27.214 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:27.213+0000 7f52f2992640 1 -- start start 2024-05-04T12:15:27.215 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:27.214+0000 7f52f2992640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f52ec0fe930 0x7f52ec19bed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:15:27.215 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:27.214+0000 7f52f2992640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f52ec1012a0 0x7f52ec19c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:15:27.215 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:27.214+0000 7f52f2992640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52ec19c9e0 con 0x7f52ec1012a0 2024-05-04T12:15:27.215 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:27.214+0000 7f52f2992640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52ec19cb50 con 0x7f52ec0fe930 2024-05-04T12:15:27.215 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:27.214+0000 7f52eb7fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f52ec1012a0 0x7f52ec19c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:15:27.215 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:27.214+0000 7f52eb7fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f52ec1012a0 0x7f52ec19c410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:51360/0 (socket says 172.21.15.79:51360) 2024-05-04T12:15:27.215 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:27.214+0000 7f52eb7fe640 1 -- 172.21.15.79:0/2803906705 learned_addr learned my addr 172.21.15.79:0/2803906705 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:15:27.215 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:27.214+0000 7f52ebfff640 1 --2- 172.21.15.79:0/2803906705 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f52ec0fe930 0x7f52ec19bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:15:27.215 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:27.214+0000 7f52eb7fe640 1 -- 172.21.15.79:0/2803906705 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f52ec0fe930 msgr2=0x7f52ec19bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:15:27.215 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:27.214+0000 7f52eb7fe640 1 --2- 172.21.15.79:0/2803906705 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f52ec0fe930 0x7f52ec19bed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:27.215 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:27.214+0000 7f52eb7fe640 1 -- 172.21.15.79:0/2803906705 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52e0009560 con 0x7f52ec1012a0 2024-05-04T12:15:27.215 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:27.214+0000 7f52eb7fe640 1 --2- 172.21.15.79:0/2803906705 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f52ec1012a0 0x7f52ec19c410 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f52e0009230 tx=0x7f52e0004f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:15:27.215 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:27.214+0000 7f52e97fa640 1 -- 172.21.15.79:0/2803906705 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f52e0004520 con 0x7f52ec1012a0 2024-05-04T12:15:27.215 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:27.214+0000 7f52e97fa640 1 -- 172.21.15.79:0/2803906705 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f52e00054b0 con 0x7f52ec1012a0 2024-05-04T12:15:27.216 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:27.214+0000 7f52f2992640 1 -- 172.21.15.79:0/2803906705 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52ec1a1580 con 0x7f52ec1012a0 2024-05-04T12:15:27.216 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:27.214+0000 7f52e97fa640 1 -- 172.21.15.79:0/2803906705 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f52e0034440 con 0x7f52ec1012a0 2024-05-04T12:15:27.216 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:27.214+0000 7f52f2992640 1 -- 172.21.15.79:0/2803906705 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52ec1a1a80 con 0x7f52ec1012a0 2024-05-04T12:15:27.216 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:27.215+0000 7f52f2992640 1 -- 172.21.15.79:0/2803906705 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52ec196000 con 0x7f52ec1012a0 2024-05-04T12:15:27.216 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:27.216+0000 7f52e97fa640 1 -- 172.21.15.79:0/2803906705 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f52e00037f0 con 0x7f52ec1012a0 2024-05-04T12:15:27.217 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:27.216+0000 7f52e97fa640 1 --2- 172.21.15.79:0/2803906705 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f52d406c510 0x7f52d406e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:15:27.217 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:27.216+0000 7f52e97fa640 1 -- 172.21.15.79:0/2803906705 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f52e00adc50 con 0x7f52ec1012a0 2024-05-04T12:15:27.217 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:27.216+0000 7f52ebfff640 1 --2- 172.21.15.79:0/2803906705 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f52d406c510 0x7f52d406e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:15:27.217 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:27.217+0000 7f52ebfff640 1 --2- 172.21.15.79:0/2803906705 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f52d406c510 0x7f52d406e9d0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f52dc00bbe0 tx=0x7f52dc00b390 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:15:27.220 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:27.221+0000 7f52e97fa640 1 -- 172.21.15.79:0/2803906705 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f52e0078500 con 0x7f52ec1012a0 2024-05-04T12:15:27.397 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:27.398+0000 7f52f2992640 1 -- 172.21.15.79:0/2803906705 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f52ec02c4e0 con 0x7f52ec1012a0 2024-05-04T12:15:27.398 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:27.399+0000 7f52e97fa640 1 -- 172.21.15.79:0/2803906705 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v46) v1 ==== 74+0+15150 (secure 0 0 0) 0x7f52e007bb20 con 0x7f52ec1012a0 2024-05-04T12:15:27.399 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-05-04T12:15:27.399 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":46,"fsid":"d9b985c4-0a0e-11ef-bc97-c7b262605968","created":"2024-05-04T12:07:51.285061+0000","modified":"2024-05-04T12:15:15.634292+0000","last_up_change":"2024-05-04T12:13:43.138775+0000","last_in_change":"2024-05-04T12:13:23.351367+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-04T12:12:27.893460+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs.cephfs.meta","create_time":"2024-05-04T12:14:58.637565+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.690000057220459,"score_stable":1.690000057220459,"optimal_score":1,"raw_score_acting":1.690000057220459,"raw_score_stable":1.690000057220459,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.cephfs.data","create_time":"2024-05-04T12:14:59.315726+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"904ab939-d572-43db-8c6f-fb309334b478","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6802","nonce":1309201220},{"type":"v1","addr":"172.21.15.79:6803","nonce":1309201220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6804","nonce":1309201220},{"type":"v1","addr":"172.21.15.79:6805","nonce":1309201220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6808","nonce":1309201220},{"type":"v1","addr":"172.21.15.79:6809","nonce":1309201220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6806","nonce":1309201220},{"type":"v1","addr":"172.21.15.79:6807","nonce":1309201220}]},"public_addr":"172.21.15.79:6803/1309201220","cluster_addr":"172.21.15.79:6805/1309201220","heartbeat_back_addr":"172.21.15.79:6809/1309201220","heartbeat_front_addr":"172.21.15.79:6807/1309201220","state":["exists","up"]},{"osd":1,"uuid":"bce71499-a9a8-4d8f-932e-1c85918eee61","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6810","nonce":3443379349},{"type":"v1","addr":"172.21.15.79:6811","nonce":3443379349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6812","nonce":3443379349},{"type":"v1","addr":"172.21.15.79:6813","nonce":3443379349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6816","nonce":3443379349},{"type":"v1","addr":"172.21.15.79:6817","nonce":3443379349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6814","nonce":3443379349},{"type":"v1","addr":"172.21.15.79:6815","nonce":3443379349}]},"public_addr":"172.21.15.79:6811/3443379349","cluster_addr":"172.21.15.79:6813/3443379349","heartbeat_back_addr":"172.21.15.79:6817/3443379349","heartbeat_front_addr":"172.21.15.79:6815/3443379349","state":["exists","up"]},{"osd":2,"uuid":"cfb80418-73e3-4001-8d30-53f06365d435","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6818","nonce":3776007511},{"type":"v1","addr":"172.21.15.79:6819","nonce":3776007511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6820","nonce":3776007511},{"type":"v1","addr":"172.21.15.79:6821","nonce":3776007511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6824","nonce":3776007511},{"type":"v1","addr":"172.21.15.79:6825","nonce":3776007511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6822","nonce":3776007511},{"type":"v1","addr":"172.21.15.79:6823","nonce":3776007511}]},"public_addr":"172.21.15.79:6819/3776007511","cluster_addr":"172.21.15.79:6821/3776007511","heartbeat_back_addr":"172.21.15.79:6825/3776007511","heartbeat_front_addr":"172.21.15.79:6823/3776007511","state":["exists","up"]},{"osd":3,"uuid":"67d66fd8-d092-4ba0-8418-799689ae0a04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":1187278645},{"type":"v1","addr":"172.21.15.146:6801","nonce":1187278645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":1187278645},{"type":"v1","addr":"172.21.15.146:6803","nonce":1187278645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":1187278645},{"type":"v1","addr":"172.21.15.146:6807","nonce":1187278645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":1187278645},{"type":"v1","addr":"172.21.15.146:6805","nonce":1187278645}]},"public_addr":"172.21.15.146:6801/1187278645","cluster_addr":"172.21.15.146:6803/1187278645","heartbeat_back_addr":"172.21.15.146:6807/1187278645","heartbeat_front_addr":"172.21.15.146:6805/1187278645","state":["exists","up"]},{"osd":4,"uuid":"845a6602-d767-45c8-9fd1-1cbbe89adb23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":3913803877},{"type":"v1","addr":"172.21.15.146:6809","nonce":3913803877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":3913803877},{"type":"v1","addr":"172.21.15.146:6811","nonce":3913803877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":3913803877},{"type":"v1","addr":"172.21.15.146:6815","nonce":3913803877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":3913803877},{"type":"v1","addr":"172.21.15.146:6813","nonce":3913803877}]},"public_addr":"172.21.15.146:6809/3913803877","cluster_addr":"172.21.15.146:6811/3913803877","heartbeat_back_addr":"172.21.15.146:6815/3913803877","heartbeat_front_addr":"172.21.15.146:6813/3913803877","state":["exists","up"]},{"osd":5,"uuid":"02174a1e-ba23-4235-bd22-0969c319b052","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":3191460146},{"type":"v1","addr":"172.21.15.146:6817","nonce":3191460146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":3191460146},{"type":"v1","addr":"172.21.15.146:6819","nonce":3191460146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":3191460146},{"type":"v1","addr":"172.21.15.146:6823","nonce":3191460146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3191460146},{"type":"v1","addr":"172.21.15.146:6821","nonce":3191460146}]},"public_addr":"172.21.15.146:6817/3191460146","cluster_addr":"172.21.15.146:6819/3191460146","heartbeat_back_addr":"172.21.15.146:6823/3191460146","heartbeat_front_addr":"172.21.15.146:6821/3191460146","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-04T12:11:26.960942+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-04T12:11:56.001694+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-04T12:12:24.349808+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-04T12:12:48.199923+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-04T12:13:14.153595+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-04T12:13:41.185784+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.79:6827/4053411185":"2024-05-05T12:15:15.634271+0000","172.21.15.79:6800/1620705054":"2024-05-05T12:08:58.217661+0000","172.21.15.79:0/3859814994":"2024-05-05T12:08:26.031771+0000","172.21.15.79:6800/839275478":"2024-05-05T12:08:26.031771+0000","172.21.15.79:0/2668033337":"2024-05-05T12:08:26.031771+0000","172.21.15.79:6801/1620705054":"2024-05-05T12:08:58.217661+0000","172.21.15.79:0/2735058954":"2024-05-05T12:08:26.031771+0000","172.21.15.79:0/1765481553":"2024-05-05T12:08:58.217661+0000","172.21.15.79:0/865611036":"2024-05-05T12:10:11.585392+0000","172.21.15.79:0/3218172085":"2024-05-05T12:08:58.217661+0000","172.21.15.79:0/4109386639":"2024-05-05T12:08:58.217661+0000","172.21.15.79:6800/4263681324":"2024-05-05T12:10:11.585392+0000","172.21.15.79:6801/4263681324":"2024-05-05T12:10:11.585392+0000","172.21.15.79:0/1526054010":"2024-05-05T12:10:11.585392+0000","172.21.15.79:6801/839275478":"2024-05-05T12:08:26.031771+0000","172.21.15.79:0/488603041":"2024-05-05T12:10:11.585392+0000","172.21.15.79:6826/4053411185":"2024-05-05T12:15:15.634271+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-04T12:15:27.401 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:27.401+0000 7f52f2992640 1 -- 172.21.15.79:0/2803906705 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f52d406c510 msgr2=0x7f52d406e9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:15:27.401 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:27.401+0000 7f52f2992640 1 --2- 172.21.15.79:0/2803906705 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f52d406c510 0x7f52d406e9d0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f52dc00bbe0 tx=0x7f52dc00b390 comp rx=0 tx=0).stop 2024-05-04T12:15:27.401 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:27.401+0000 7f52f2992640 1 -- 172.21.15.79:0/2803906705 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f52ec1012a0 msgr2=0x7f52ec19c410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:15:27.401 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:27.401+0000 7f52f2992640 1 --2- 172.21.15.79:0/2803906705 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f52ec1012a0 0x7f52ec19c410 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f52e0009230 tx=0x7f52e0004f00 comp rx=0 tx=0).stop 2024-05-04T12:15:27.401 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:27.401+0000 7f52f2992640 1 -- 172.21.15.79:0/2803906705 shutdown_connections 2024-05-04T12:15:27.401 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:27.401+0000 7f52f2992640 1 --2- 172.21.15.79:0/2803906705 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f52d406c510 0x7f52d406e9d0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:27.401 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:27.401+0000 7f52f2992640 1 --2- 172.21.15.79:0/2803906705 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f52ec1012a0 0x7f52ec19c410 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:27.401 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:27.401+0000 7f52f2992640 1 --2- 172.21.15.79:0/2803906705 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f52ec0fe930 0x7f52ec19bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:27.401 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:27.401+0000 7f52f2992640 1 -- 172.21.15.79:0/2803906705 >> 172.21.15.79:0/2803906705 conn(0x7f52ec0fa530 msgr2=0x7f52ec0fc970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:15:27.401 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:27.401+0000 7f52f2992640 1 -- 172.21.15.79:0/2803906705 shutdown_connections 2024-05-04T12:15:27.401 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:27.401+0000 7f52f2992640 1 -- 172.21.15.79:0/2803906705 wait complete. 2024-05-04T12:15:27.413 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-04T12:12:27.893460+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 6, 'score_stable': 6, 'optimal_score': 0.5, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs.cephfs.meta', 'create_time': '2024-05-04T12:14:58.637565+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '44', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.690000057220459, 'score_stable': 1.690000057220459, 'optimal_score': 1, 'raw_score_acting': 1.690000057220459, 'raw_score_stable': 1.690000057220459, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs.cephfs.data', 'create_time': '2024-05-04T12:14:59.315726+0000', 'flags': 131073, 'flags_names': 'hashpspool,bulk', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-04T12:15:27.413 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-04T12:15:27.523 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:27.520+0000 7f9ff777a640 1 -- 172.21.15.79:0/2162058582 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9ff0103460 msgr2=0x7f9ff0105850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:15:27.523 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:27.520+0000 7f9ff777a640 1 --2- 172.21.15.79:0/2162058582 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9ff0103460 0x7f9ff0105850 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f9fe00098b0 tx=0x7f9fe002bcd0 comp rx=0 tx=0).stop 2024-05-04T12:15:27.523 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:27.520+0000 7f9ff777a640 1 -- 172.21.15.79:0/2162058582 shutdown_connections 2024-05-04T12:15:27.523 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:27.520+0000 7f9ff777a640 1 --2- 172.21.15.79:0/2162058582 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9ff0103460 0x7f9ff0105850 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:27.523 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:27.520+0000 7f9ff777a640 1 --2- 172.21.15.79:0/2162058582 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9ff0100b30 0x7f9ff0102f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:27.523 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:27.521+0000 7f9ff777a640 1 -- 172.21.15.79:0/2162058582 >> 172.21.15.79:0/2162058582 conn(0x7f9ff00fa4e0 msgr2=0x7f9ff00fc960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:15:27.523 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:27.521+0000 7f9ff777a640 1 -- 172.21.15.79:0/2162058582 shutdown_connections 2024-05-04T12:15:27.523 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:27.521+0000 7f9ff777a640 1 -- 172.21.15.79:0/2162058582 wait complete. 2024-05-04T12:15:27.524 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:27.521+0000 7f9ff777a640 1 Processor -- start 2024-05-04T12:15:27.524 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:27.521+0000 7f9ff777a640 1 -- start start 2024-05-04T12:15:27.524 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:27.521+0000 7f9ff777a640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9ff0100b30 0x7f9ff01004a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:15:27.524 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:27.521+0000 7f9ff777a640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9ff0103460 0x7f9ff00feae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:15:27.524 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:27.521+0000 7f9ff777a640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ff00ff020 con 0x7f9ff0100b30 2024-05-04T12:15:27.524 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:27.521+0000 7f9ff777a640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ff00ff190 con 0x7f9ff0103460 2024-05-04T12:15:27.524 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:27.521+0000 7f9ff5516640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9ff0100b30 0x7f9ff01004a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:15:27.524 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:27.521+0000 7f9ff5516640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9ff0100b30 0x7f9ff01004a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:51388/0 (socket says 172.21.15.79:51388) 2024-05-04T12:15:27.524 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:27.521+0000 7f9ff5516640 1 -- 172.21.15.79:0/1414780509 learned_addr learned my addr 172.21.15.79:0/1414780509 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:15:27.524 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:27.522+0000 7f9ff4d15640 1 --2- 172.21.15.79:0/1414780509 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9ff0103460 0x7f9ff00feae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:15:27.524 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:27.522+0000 7f9ff5516640 1 -- 172.21.15.79:0/1414780509 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9ff0103460 msgr2=0x7f9ff00feae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:15:27.524 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:27.522+0000 7f9ff5516640 1 --2- 172.21.15.79:0/1414780509 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9ff0103460 0x7f9ff00feae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:27.525 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:27.522+0000 7f9ff5516640 1 -- 172.21.15.79:0/1414780509 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fe0009560 con 0x7f9ff0100b30 2024-05-04T12:15:27.525 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:27.522+0000 7f9ff5516640 1 --2- 172.21.15.79:0/1414780509 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9ff0100b30 0x7f9ff01004a0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f9fec00d8d0 tx=0x7f9fec00dd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:15:27.525 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:27.522+0000 7f9fe67fc640 1 -- 172.21.15.79:0/1414780509 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9fec009c00 con 0x7f9ff0100b30 2024-05-04T12:15:27.525 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:27.522+0000 7f9fe67fc640 1 -- 172.21.15.79:0/1414780509 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f9fec0044d0 con 0x7f9ff0100b30 2024-05-04T12:15:27.525 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:27.522+0000 7f9fe67fc640 1 -- 172.21.15.79:0/1414780509 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9fec0049d0 con 0x7f9ff0100b30 2024-05-04T12:15:27.525 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:27.522+0000 7f9ff777a640 1 -- 172.21.15.79:0/1414780509 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ff00ff470 con 0x7f9ff0100b30 2024-05-04T12:15:27.525 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:27.522+0000 7f9ff777a640 1 -- 172.21.15.79:0/1414780509 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ff019da30 con 0x7f9ff0100b30 2024-05-04T12:15:27.525 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:27.523+0000 7f9ff777a640 1 -- 172.21.15.79:0/1414780509 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ff0108c90 con 0x7f9ff0100b30 2024-05-04T12:15:27.525 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:27.524+0000 7f9fe67fc640 1 -- 172.21.15.79:0/1414780509 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f9fec004020 con 0x7f9ff0100b30 2024-05-04T12:15:27.526 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:27.524+0000 7f9fe67fc640 1 --2- 172.21.15.79:0/1414780509 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f9fdc06c450 0x7f9fdc06e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:15:27.526 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:27.524+0000 7f9fe67fc640 1 -- 172.21.15.79:0/1414780509 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f9fec08b530 con 0x7f9ff0100b30 2024-05-04T12:15:27.526 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:27.524+0000 7f9ff4d15640 1 --2- 172.21.15.79:0/1414780509 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f9fdc06c450 0x7f9fdc06e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:15:27.526 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:27.524+0000 7f9ff4d15640 1 --2- 172.21.15.79:0/1414780509 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f9fdc06c450 0x7f9fdc06e910 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f9fe0036040 tx=0x7f9fe0005fd0 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:15:27.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:27 smithi146 ceph-mon[35311]: pgmap v191: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 8.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-05-04T12:15:27.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:27 smithi146 ceph-mon[35311]: from='client.? 172.21.15.79:0/1058536457' entity='client.admin' cmd=[{"prefix": "mds versions", "format": "json"}]: dispatch 2024-05-04T12:15:27.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:27 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:15:27.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:27 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-04T12:15:27.527 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:27.528+0000 7f9fe67fc640 1 -- 172.21.15.79:0/1414780509 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f9fec055d30 con 0x7f9ff0100b30 2024-05-04T12:15:27.704 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:27.704+0000 7f9ff777a640 1 -- 172.21.15.79:0/1414780509 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f9ff019dbc0 con 0x7f9ff0100b30 2024-05-04T12:15:27.704 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:27.704+0000 7f9fe67fc640 1 -- 172.21.15.79:0/1414780509 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v46) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9fec059350 con 0x7f9ff0100b30 2024-05-04T12:15:27.704 INFO:teuthology.orchestra.run.smithi079.stdout:pg_num: 1 2024-05-04T12:15:27.706 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:27.706+0000 7f9ff777a640 1 -- 172.21.15.79:0/1414780509 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f9fdc06c450 msgr2=0x7f9fdc06e910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:15:27.706 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:27.706+0000 7f9ff777a640 1 --2- 172.21.15.79:0/1414780509 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f9fdc06c450 0x7f9fdc06e910 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f9fe0036040 tx=0x7f9fe0005fd0 comp rx=0 tx=0).stop 2024-05-04T12:15:27.706 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:27.706+0000 7f9ff777a640 1 -- 172.21.15.79:0/1414780509 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9ff0100b30 msgr2=0x7f9ff01004a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:15:27.706 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:27.706+0000 7f9ff777a640 1 --2- 172.21.15.79:0/1414780509 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9ff0100b30 0x7f9ff01004a0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f9fec00d8d0 tx=0x7f9fec00dd90 comp rx=0 tx=0).stop 2024-05-04T12:15:27.707 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:27.706+0000 7f9ff777a640 1 -- 172.21.15.79:0/1414780509 shutdown_connections 2024-05-04T12:15:27.707 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:27.706+0000 7f9ff777a640 1 --2- 172.21.15.79:0/1414780509 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f9fdc06c450 0x7f9fdc06e910 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:27.707 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:27.706+0000 7f9ff777a640 1 --2- 172.21.15.79:0/1414780509 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9ff0103460 0x7f9ff00feae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:27.707 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:27.706+0000 7f9ff777a640 1 --2- 172.21.15.79:0/1414780509 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9ff0100b30 0x7f9ff01004a0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:27.707 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:27.706+0000 7f9ff777a640 1 -- 172.21.15.79:0/1414780509 >> 172.21.15.79:0/1414780509 conn(0x7f9ff00fa4e0 msgr2=0x7f9ff00fc960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:15:27.707 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:27.706+0000 7f9ff777a640 1 -- 172.21.15.79:0/1414780509 shutdown_connections 2024-05-04T12:15:27.707 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:27.707+0000 7f9ff777a640 1 -- 172.21.15.79:0/1414780509 wait complete. 2024-05-04T12:15:27.716 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.meta pg_num 2024-05-04T12:15:27.818 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:27.817+0000 7f3c97bc5640 1 -- 172.21.15.79:0/4061074600 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3c900fefc0 msgr2=0x7f3c900ff3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:15:27.818 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:27.817+0000 7f3c97bc5640 1 --2- 172.21.15.79:0/4061074600 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3c900fefc0 0x7f3c900ff3e0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f3c8c009880 tx=0x7f3c8c02bce0 comp rx=0 tx=0).stop 2024-05-04T12:15:27.818 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:27.817+0000 7f3c97bc5640 1 -- 172.21.15.79:0/4061074600 shutdown_connections 2024-05-04T12:15:27.819 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:27.817+0000 7f3c97bc5640 1 --2- 172.21.15.79:0/4061074600 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3c901001b0 0x7f3c90100630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:27.819 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:27.817+0000 7f3c97bc5640 1 --2- 172.21.15.79:0/4061074600 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3c900fefc0 0x7f3c900ff3e0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:27.819 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:27.817+0000 7f3c97bc5640 1 -- 172.21.15.79:0/4061074600 >> 172.21.15.79:0/4061074600 conn(0x7f3c900fa510 msgr2=0x7f3c900fc970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:15:27.819 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:27.817+0000 7f3c97bc5640 1 -- 172.21.15.79:0/4061074600 shutdown_connections 2024-05-04T12:15:27.819 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:27.817+0000 7f3c97bc5640 1 -- 172.21.15.79:0/4061074600 wait complete. 2024-05-04T12:15:27.819 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:27.818+0000 7f3c97bc5640 1 Processor -- start 2024-05-04T12:15:27.819 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:27.818+0000 7f3c97bc5640 1 -- start start 2024-05-04T12:15:27.819 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:27.818+0000 7f3c97bc5640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3c900fefc0 0x7f3c901937e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:15:27.819 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:27.818+0000 7f3c97bc5640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3c901001b0 0x7f3c90193d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:15:27.819 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:27.818+0000 7f3c97bc5640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c901942f0 con 0x7f3c901001b0 2024-05-04T12:15:27.819 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:27.818+0000 7f3c97bc5640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c90194460 con 0x7f3c900fefc0 2024-05-04T12:15:27.819 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:27.818+0000 7f3c95160640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3c901001b0 0x7f3c90193d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:15:27.819 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:27.818+0000 7f3c95160640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3c901001b0 0x7f3c90193d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:51406/0 (socket says 172.21.15.79:51406) 2024-05-04T12:15:27.820 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:27.818+0000 7f3c95160640 1 -- 172.21.15.79:0/1173005205 learned_addr learned my addr 172.21.15.79:0/1173005205 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:15:27.820 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:27.818+0000 7f3c95961640 1 --2- 172.21.15.79:0/1173005205 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3c900fefc0 0x7f3c901937e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:15:27.820 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:27.818+0000 7f3c95160640 1 -- 172.21.15.79:0/1173005205 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3c900fefc0 msgr2=0x7f3c901937e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:15:27.820 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:27.818+0000 7f3c95160640 1 --2- 172.21.15.79:0/1173005205 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3c900fefc0 0x7f3c901937e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:27.820 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:27.818+0000 7f3c95160640 1 -- 172.21.15.79:0/1173005205 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c8c009560 con 0x7f3c901001b0 2024-05-04T12:15:27.820 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:27.818+0000 7f3c95160640 1 --2- 172.21.15.79:0/1173005205 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3c901001b0 0x7f3c90193d20 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f3c8000b420 tx=0x7f3c8000b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:15:27.820 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:27.818+0000 7f3c86ffd640 1 -- 172.21.15.79:0/1173005205 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3c80011800 con 0x7f3c901001b0 2024-05-04T12:15:27.820 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:27.818+0000 7f3c86ffd640 1 -- 172.21.15.79:0/1173005205 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f3c80011e10 con 0x7f3c901001b0 2024-05-04T12:15:27.820 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:27.818+0000 7f3c97bc5640 1 -- 172.21.15.79:0/1173005205 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c90198ef0 con 0x7f3c901001b0 2024-05-04T12:15:27.820 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:27.818+0000 7f3c86ffd640 1 -- 172.21.15.79:0/1173005205 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3c8000f2e0 con 0x7f3c901001b0 2024-05-04T12:15:27.820 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:27.818+0000 7f3c97bc5640 1 -- 172.21.15.79:0/1173005205 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c901994a0 con 0x7f3c901001b0 2024-05-04T12:15:27.821 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:27.819+0000 7f3c97bc5640 1 -- 172.21.15.79:0/1173005205 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c90106b10 con 0x7f3c901001b0 2024-05-04T12:15:27.821 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:27.820+0000 7f3c86ffd640 1 -- 172.21.15.79:0/1173005205 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f3c80011960 con 0x7f3c901001b0 2024-05-04T12:15:27.821 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:27.820+0000 7f3c86ffd640 1 --2- 172.21.15.79:0/1173005205 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f3c7c06e660 0x7f3c7c070b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:15:27.821 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:27.820+0000 7f3c86ffd640 1 -- 172.21.15.79:0/1173005205 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f3c8008b270 con 0x7f3c901001b0 2024-05-04T12:15:27.821 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:27.820+0000 7f3c95961640 1 --2- 172.21.15.79:0/1173005205 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f3c7c06e660 0x7f3c7c070b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:15:27.822 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:27.820+0000 7f3c95961640 1 --2- 172.21.15.79:0/1173005205 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f3c7c06e660 0x7f3c7c070b20 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f3c8c006010 tx=0x7f3c8c000bc0 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:15:27.823 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:27.824+0000 7f3c86ffd640 1 -- 172.21.15.79:0/1173005205 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f3c800559f0 con 0x7f3c901001b0 2024-05-04T12:15:28.008 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:28.008+0000 7f3c97bc5640 1 -- 172.21.15.79:0/1173005205 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"} v 0) v1 -- 0x7f3c90199720 con 0x7f3c901001b0 2024-05-04T12:15:28.008 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:28.009+0000 7f3c86ffd640 1 -- 172.21.15.79:0/1173005205 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]=0 v46) v1 ==== 107+0+11 (secure 0 0 0) 0x7f3c80059010 con 0x7f3c901001b0 2024-05-04T12:15:28.008 INFO:teuthology.orchestra.run.smithi079.stdout:pg_num: 32 2024-05-04T12:15:28.010 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:28.011+0000 7f3c97bc5640 1 -- 172.21.15.79:0/1173005205 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f3c7c06e660 msgr2=0x7f3c7c070b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:15:28.010 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:28.011+0000 7f3c97bc5640 1 --2- 172.21.15.79:0/1173005205 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f3c7c06e660 0x7f3c7c070b20 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f3c8c006010 tx=0x7f3c8c000bc0 comp rx=0 tx=0).stop 2024-05-04T12:15:28.011 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:28.011+0000 7f3c97bc5640 1 -- 172.21.15.79:0/1173005205 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3c901001b0 msgr2=0x7f3c90193d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:15:28.011 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:28.011+0000 7f3c97bc5640 1 --2- 172.21.15.79:0/1173005205 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3c901001b0 0x7f3c90193d20 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f3c8000b420 tx=0x7f3c8000b8e0 comp rx=0 tx=0).stop 2024-05-04T12:15:28.011 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:28.011+0000 7f3c97bc5640 1 -- 172.21.15.79:0/1173005205 shutdown_connections 2024-05-04T12:15:28.011 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:28.011+0000 7f3c97bc5640 1 --2- 172.21.15.79:0/1173005205 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f3c7c06e660 0x7f3c7c070b20 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:28.011 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:28.011+0000 7f3c97bc5640 1 --2- 172.21.15.79:0/1173005205 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3c901001b0 0x7f3c90193d20 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:28.011 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:28.011+0000 7f3c97bc5640 1 --2- 172.21.15.79:0/1173005205 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3c900fefc0 0x7f3c901937e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:28.011 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:28.011+0000 7f3c97bc5640 1 -- 172.21.15.79:0/1173005205 >> 172.21.15.79:0/1173005205 conn(0x7f3c900fa510 msgr2=0x7f3c901033e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:15:28.011 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:28.011+0000 7f3c97bc5640 1 -- 172.21.15.79:0/1173005205 shutdown_connections 2024-05-04T12:15:28.011 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:28.011+0000 7f3c97bc5640 1 -- 172.21.15.79:0/1173005205 wait complete. 2024-05-04T12:15:28.021 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.data pg_num 2024-05-04T12:15:28.137 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:28.136+0000 7f78b9c7d640 1 -- 172.21.15.79:0/3547893102 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f78b4103480 msgr2=0x7f78b4105870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:15:28.138 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:28.136+0000 7f78b9c7d640 1 --2- 172.21.15.79:0/3547893102 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f78b4103480 0x7f78b4105870 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f78a8009880 tx=0x7f78a802bce0 comp rx=0 tx=0).stop 2024-05-04T12:15:28.138 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:28.136+0000 7f78b9c7d640 1 -- 172.21.15.79:0/3547893102 shutdown_connections 2024-05-04T12:15:28.138 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:28.136+0000 7f78b9c7d640 1 --2- 172.21.15.79:0/3547893102 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f78b4103480 0x7f78b4105870 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:28.138 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:28.136+0000 7f78b9c7d640 1 --2- 172.21.15.79:0/3547893102 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f78b4100b50 0x7f78b4102f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:28.138 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:28.136+0000 7f78b9c7d640 1 -- 172.21.15.79:0/3547893102 >> 172.21.15.79:0/3547893102 conn(0x7f78b40fa4e0 msgr2=0x7f78b40fc940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:15:28.138 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:28.136+0000 7f78b9c7d640 1 -- 172.21.15.79:0/3547893102 shutdown_connections 2024-05-04T12:15:28.138 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:28.136+0000 7f78b9c7d640 1 -- 172.21.15.79:0/3547893102 wait complete. 2024-05-04T12:15:28.138 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:28.137+0000 7f78b9c7d640 1 Processor -- start 2024-05-04T12:15:28.138 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:28.137+0000 7f78b9c7d640 1 -- start start 2024-05-04T12:15:28.138 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:28.137+0000 7f78b9c7d640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f78b4100b50 0x7f78b41959a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:15:28.138 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:28.137+0000 7f78b9c7d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f78b4103480 0x7f78b4195ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:15:28.138 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:28.137+0000 7f78b9c7d640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78b41964b0 con 0x7f78b4103480 2024-05-04T12:15:28.139 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:28.137+0000 7f78b9c7d640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78b4196620 con 0x7f78b4100b50 2024-05-04T12:15:28.139 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:28.137+0000 7f78b2ffd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f78b4103480 0x7f78b4195ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:15:28.139 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:28.137+0000 7f78b2ffd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f78b4103480 0x7f78b4195ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:51424/0 (socket says 172.21.15.79:51424) 2024-05-04T12:15:28.139 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:28.137+0000 7f78b2ffd640 1 -- 172.21.15.79:0/2280532527 learned_addr learned my addr 172.21.15.79:0/2280532527 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:15:28.139 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:28.137+0000 7f78b37fe640 1 --2- 172.21.15.79:0/2280532527 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f78b4100b50 0x7f78b41959a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:15:28.139 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:28.138+0000 7f78b2ffd640 1 -- 172.21.15.79:0/2280532527 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f78b4100b50 msgr2=0x7f78b41959a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:15:28.139 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:28.138+0000 7f78b2ffd640 1 --2- 172.21.15.79:0/2280532527 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f78b4100b50 0x7f78b41959a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:28.139 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:28.138+0000 7f78b2ffd640 1 -- 172.21.15.79:0/2280532527 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78a8009560 con 0x7f78b4103480 2024-05-04T12:15:28.139 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:28.138+0000 7f78b2ffd640 1 --2- 172.21.15.79:0/2280532527 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f78b4103480 0x7f78b4195ee0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f78a8009230 tx=0x7f78a8004e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:15:28.139 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:28.138+0000 7f78b0ff9640 1 -- 172.21.15.79:0/2280532527 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f78a8005000 con 0x7f78b4103480 2024-05-04T12:15:28.139 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:28.138+0000 7f78b9c7d640 1 -- 172.21.15.79:0/2280532527 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78b419b050 con 0x7f78b4103480 2024-05-04T12:15:28.139 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:28.138+0000 7f78b0ff9640 1 -- 172.21.15.79:0/2280532527 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f78a8005610 con 0x7f78b4103480 2024-05-04T12:15:28.139 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:28.138+0000 7f78b0ff9640 1 -- 172.21.15.79:0/2280532527 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f78a8034440 con 0x7f78b4103480 2024-05-04T12:15:28.139 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:28.138+0000 7f78b9c7d640 1 -- 172.21.15.79:0/2280532527 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78b419b550 con 0x7f78b4103480 2024-05-04T12:15:28.140 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:28.139+0000 7f78b9c7d640 1 -- 172.21.15.79:0/2280532527 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78b4108dd0 con 0x7f78b4103480 2024-05-04T12:15:28.140 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:28.140+0000 7f78b0ff9640 1 -- 172.21.15.79:0/2280532527 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f78a8003880 con 0x7f78b4103480 2024-05-04T12:15:28.140 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:28.140+0000 7f78b0ff9640 1 --2- 172.21.15.79:0/2280532527 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f789c06c450 0x7f789c06e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:15:28.141 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:28.140+0000 7f78b0ff9640 1 -- 172.21.15.79:0/2280532527 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f78a80ade20 con 0x7f78b4103480 2024-05-04T12:15:28.141 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:28.140+0000 7f78b37fe640 1 --2- 172.21.15.79:0/2280532527 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f789c06c450 0x7f789c06e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:15:28.141 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:28.141+0000 7f78b37fe640 1 --2- 172.21.15.79:0/2280532527 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f789c06c450 0x7f789c06e910 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f78a4005fd0 tx=0x7f78a4005850 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:15:28.144 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:28.145+0000 7f78b0ff9640 1 -- 172.21.15.79:0/2280532527 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f78a80785a0 con 0x7f78b4103480 2024-05-04T12:15:28.270 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:28 smithi079 ceph-mon[27222]: from='client.? 172.21.15.79:0/2803906705' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-04T12:15:28.270 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:28 smithi079 ceph-mon[27222]: from='client.? 172.21.15.79:0/1414780509' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-04T12:15:28.270 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:28 smithi079 ceph-mon[27222]: from='client.? 172.21.15.79:0/1173005205' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-05-04T12:15:28.277 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:28 smithi146 ceph-mon[35311]: from='client.? 172.21.15.79:0/2803906705' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-04T12:15:28.277 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:28 smithi146 ceph-mon[35311]: from='client.? 172.21.15.79:0/1414780509' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-04T12:15:28.277 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:28 smithi146 ceph-mon[35311]: from='client.? 172.21.15.79:0/1173005205' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-05-04T12:15:28.328 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:28.329+0000 7f78b9c7d640 1 -- 172.21.15.79:0/2280532527 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"} v 0) v1 -- 0x7f78b4108f60 con 0x7f78b4103480 2024-05-04T12:15:28.328 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:28.329+0000 7f78b0ff9640 1 -- 172.21.15.79:0/2280532527 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]=0 v46) v1 ==== 107+0+11 (secure 0 0 0) 0x7f78b4108f60 con 0x7f78b4103480 2024-05-04T12:15:28.328 INFO:teuthology.orchestra.run.smithi079.stdout:pg_num: 32 2024-05-04T12:15:28.331 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:28.331+0000 7f78b9c7d640 1 -- 172.21.15.79:0/2280532527 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f789c06c450 msgr2=0x7f789c06e910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:15:28.331 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:28.331+0000 7f78b9c7d640 1 --2- 172.21.15.79:0/2280532527 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f789c06c450 0x7f789c06e910 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f78a4005fd0 tx=0x7f78a4005850 comp rx=0 tx=0).stop 2024-05-04T12:15:28.331 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:28.331+0000 7f78b9c7d640 1 -- 172.21.15.79:0/2280532527 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f78b4103480 msgr2=0x7f78b4195ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:15:28.331 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:28.331+0000 7f78b9c7d640 1 --2- 172.21.15.79:0/2280532527 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f78b4103480 0x7f78b4195ee0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f78a8009230 tx=0x7f78a8004e30 comp rx=0 tx=0).stop 2024-05-04T12:15:28.331 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:28.331+0000 7f78b9c7d640 1 -- 172.21.15.79:0/2280532527 shutdown_connections 2024-05-04T12:15:28.331 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:28.331+0000 7f78b9c7d640 1 --2- 172.21.15.79:0/2280532527 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f789c06c450 0x7f789c06e910 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:28.331 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:28.331+0000 7f78b9c7d640 1 --2- 172.21.15.79:0/2280532527 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f78b4103480 0x7f78b4195ee0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:28.331 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:28.331+0000 7f78b9c7d640 1 --2- 172.21.15.79:0/2280532527 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f78b4100b50 0x7f78b41959a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:28.331 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:28.331+0000 7f78b9c7d640 1 -- 172.21.15.79:0/2280532527 >> 172.21.15.79:0/2280532527 conn(0x7f78b40fa4e0 msgr2=0x7f78b40fc940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:15:28.331 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:28.331+0000 7f78b9c7d640 1 -- 172.21.15.79:0/2280532527 shutdown_connections 2024-05-04T12:15:28.331 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:28.331+0000 7f78b9c7d640 1 -- 172.21.15.79:0/2280532527 wait complete. 2024-05-04T12:15:28.341 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-04T12:15:28.452 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:28.451+0000 7f095b524640 1 -- 172.21.15.79:0/229149179 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f09541001c0 msgr2=0x7f0954100640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:15:28.452 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:28.451+0000 7f095b524640 1 --2- 172.21.15.79:0/229149179 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f09541001c0 0x7f0954100640 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f0950009880 tx=0x7f095002bce0 comp rx=0 tx=0).stop 2024-05-04T12:15:28.453 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:28.451+0000 7f095b524640 1 -- 172.21.15.79:0/229149179 shutdown_connections 2024-05-04T12:15:28.453 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:28.451+0000 7f095b524640 1 --2- 172.21.15.79:0/229149179 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f09541001c0 0x7f0954100640 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:28.453 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:28.451+0000 7f095b524640 1 --2- 172.21.15.79:0/229149179 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f09540fefd0 0x7f09540ff3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:28.453 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:28.451+0000 7f095b524640 1 -- 172.21.15.79:0/229149179 >> 172.21.15.79:0/229149179 conn(0x7f09540fa540 msgr2=0x7f09540fc980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:15:28.453 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:28.451+0000 7f095b524640 1 -- 172.21.15.79:0/229149179 shutdown_connections 2024-05-04T12:15:28.453 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:28.451+0000 7f095b524640 1 -- 172.21.15.79:0/229149179 wait complete. 2024-05-04T12:15:28.453 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:28.452+0000 7f095b524640 1 Processor -- start 2024-05-04T12:15:28.453 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:28.452+0000 7f095b524640 1 -- start start 2024-05-04T12:15:28.453 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:28.452+0000 7f095b524640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f09540fefd0 0x7f0954068f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:15:28.453 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:28.452+0000 7f095b524640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f09541001c0 0x7f095419a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:15:28.453 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:28.452+0000 7f095b524640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f095419abb0 con 0x7f09541001c0 2024-05-04T12:15:28.453 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:28.452+0000 7f095b524640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f095419acf0 con 0x7f09540fefd0 2024-05-04T12:15:28.453 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:28.452+0000 7f0958abf640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f09541001c0 0x7f095419a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:15:28.453 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:28.452+0000 7f0958abf640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f09541001c0 0x7f095419a5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:51446/0 (socket says 172.21.15.79:51446) 2024-05-04T12:15:28.454 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:28.452+0000 7f0958abf640 1 -- 172.21.15.79:0/3164138967 learned_addr learned my addr 172.21.15.79:0/3164138967 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:15:28.454 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:28.453+0000 7f09592c0640 1 --2- 172.21.15.79:0/3164138967 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f09540fefd0 0x7f0954068f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:15:28.454 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:28.453+0000 7f0958abf640 1 -- 172.21.15.79:0/3164138967 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f09540fefd0 msgr2=0x7f0954068f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:15:28.454 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:28.453+0000 7f0958abf640 1 --2- 172.21.15.79:0/3164138967 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f09540fefd0 0x7f0954068f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:28.454 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:28.453+0000 7f0958abf640 1 -- 172.21.15.79:0/3164138967 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0950009560 con 0x7f09541001c0 2024-05-04T12:15:28.454 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:28.453+0000 7f0958abf640 1 --2- 172.21.15.79:0/3164138967 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f09541001c0 0x7f095419a5e0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f0950009230 tx=0x7f0950004e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:15:28.454 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:28.453+0000 7f094a7fc640 1 -- 172.21.15.79:0/3164138967 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0950004520 con 0x7f09541001c0 2024-05-04T12:15:28.454 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:28.453+0000 7f094a7fc640 1 -- 172.21.15.79:0/3164138967 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f09500054b0 con 0x7f09541001c0 2024-05-04T12:15:28.454 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:28.453+0000 7f095b524640 1 -- 172.21.15.79:0/3164138967 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f095419af70 con 0x7f09541001c0 2024-05-04T12:15:28.454 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:28.453+0000 7f094a7fc640 1 -- 172.21.15.79:0/3164138967 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0950034670 con 0x7f09541001c0 2024-05-04T12:15:28.454 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:28.453+0000 7f095b524640 1 -- 172.21.15.79:0/3164138967 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f095419b440 con 0x7f09541001c0 2024-05-04T12:15:28.455 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:28.454+0000 7f095b524640 1 -- 172.21.15.79:0/3164138967 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0954107bd0 con 0x7f09541001c0 2024-05-04T12:15:28.455 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:28.454+0000 7f094a7fc640 1 -- 172.21.15.79:0/3164138967 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f09500038b0 con 0x7f09541001c0 2024-05-04T12:15:28.455 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:28.455+0000 7f094a7fc640 1 --2- 172.21.15.79:0/3164138967 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f094006c510 0x7f094006e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:15:28.455 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:28.455+0000 7f094a7fc640 1 -- 172.21.15.79:0/3164138967 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f09500ade00 con 0x7f09541001c0 2024-05-04T12:15:28.455 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:28.455+0000 7f09592c0640 1 --2- 172.21.15.79:0/3164138967 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f094006c510 0x7f094006e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:15:28.456 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:28.456+0000 7f09592c0640 1 --2- 172.21.15.79:0/3164138967 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f094006c510 0x7f094006e9d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f0944005fd0 tx=0x7f0944005850 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:15:28.459 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:28.459+0000 7f094a7fc640 1 -- 172.21.15.79:0/3164138967 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f0950078580 con 0x7f09541001c0 2024-05-04T12:15:28.680 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:28.681+0000 7f095b524640 1 -- 172.21.15.79:0/3164138967 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0954107e10 con 0x7f09541001c0 2024-05-04T12:15:28.681 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:28.682+0000 7f094a7fc640 1 -- 172.21.15.79:0/3164138967 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) v1 ==== 94+0+4739 (secure 0 0 0) 0x7f095007bba0 con 0x7f09541001c0 2024-05-04T12:15:28.681 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-05-04T12:15:28.681 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":14,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14492,"name":"cephfs.smithi079.pspich","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6827/732688410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6826","nonce":732688410},{"type":"v1","addr":"172.21.15.79:6827","nonce":732688410}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24271,"name":"cephfs.smithi146.tfajce","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.146:6825/4162809119","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":4162809119},{"type":"v1","addr":"172.21.15.146:6825","nonce":4162809119}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":24283,"name":"cephfs.smithi146.mkcmzm","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.146:6827/178032877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":178032877},{"type":"v1","addr":"172.21.15.146:6827","nonce":178032877}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-04T12:15:00.083118+0000","modified":"2024-05-04T12:15:18.650123+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14482},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14482":{"gid":14482,"name":"cephfs.smithi079.hzcmip","rank":0,"incarnation":11,"state":"up:active","state_seq":5,"addr":"172.21.15.79:6829/4287465339","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6828","nonce":4287465339},{"type":"v1","addr":"172.21.15.79:6829","nonce":4287465339}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-05-04T12:15:28.681 INFO:teuthology.orchestra.run.smithi079.stderr:dumped fsmap epoch 14 2024-05-04T12:15:28.684 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:28.684+0000 7f095b524640 1 -- 172.21.15.79:0/3164138967 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f094006c510 msgr2=0x7f094006e9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:15:28.684 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:28.684+0000 7f095b524640 1 --2- 172.21.15.79:0/3164138967 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f094006c510 0x7f094006e9d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f0944005fd0 tx=0x7f0944005850 comp rx=0 tx=0).stop 2024-05-04T12:15:28.684 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:28.684+0000 7f095b524640 1 -- 172.21.15.79:0/3164138967 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f09541001c0 msgr2=0x7f095419a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:15:28.684 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:28.684+0000 7f095b524640 1 --2- 172.21.15.79:0/3164138967 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f09541001c0 0x7f095419a5e0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f0950009230 tx=0x7f0950004e30 comp rx=0 tx=0).stop 2024-05-04T12:15:28.684 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:28.684+0000 7f095b524640 1 -- 172.21.15.79:0/3164138967 shutdown_connections 2024-05-04T12:15:28.684 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:28.684+0000 7f095b524640 1 --2- 172.21.15.79:0/3164138967 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f094006c510 0x7f094006e9d0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:28.684 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:28.684+0000 7f095b524640 1 --2- 172.21.15.79:0/3164138967 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f09541001c0 0x7f095419a5e0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:28.684 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:28.684+0000 7f095b524640 1 --2- 172.21.15.79:0/3164138967 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f09540fefd0 0x7f0954068f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:28.684 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:28.684+0000 7f095b524640 1 -- 172.21.15.79:0/3164138967 >> 172.21.15.79:0/3164138967 conn(0x7f09540fa540 msgr2=0x7f09541033f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:15:28.684 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:28.684+0000 7f095b524640 1 -- 172.21.15.79:0/3164138967 shutdown_connections 2024-05-04T12:15:28.684 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:28.684+0000 7f095b524640 1 -- 172.21.15.79:0/3164138967 wait complete. 2024-05-04T12:15:28.696 DEBUG:tasks.fs:fs fscid=1,name=cephfs state = {'epoch': 14, 'max_mds': 1, 'flags': 18} 2024-05-04T12:15:28.696 INFO:teuthology.run_tasks:Running task kclient... 2024-05-04T12:15:28.719 INFO:tasks.kclient:Mounting kernel clients... 2024-05-04T12:15:28.719 INFO:tasks.kclient:config is {'client.0': {}, 'client.1': {}} 2024-05-04T12:15:28.719 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-04T12:15:28.719 DEBUG:teuthology.orchestra.run.smithi079:> ip netns list 2024-05-04T12:15:28.735 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-04T12:15:28.735 DEBUG:teuthology.orchestra.run.smithi079:> sudo ip link delete ceph-brx 2024-05-04T12:15:28.801 INFO:teuthology.orchestra.run.smithi079.stderr:Cannot find device "ceph-brx" 2024-05-04T12:15:28.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-04T12:15:28.802 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-04T12:15:28.802 DEBUG:teuthology.orchestra.run.smithi146:> ip netns list 2024-05-04T12:15:28.822 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-04T12:15:28.822 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip link delete ceph-brx 2024-05-04T12:15:28.895 INFO:teuthology.orchestra.run.smithi146.stderr:Cannot find device "ceph-brx" 2024-05-04T12:15:28.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-04T12:15:28.897 INFO:tasks.kclient:client.0 config is {'syntax': 'v1'} 2024-05-04T12:15:28.897 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-05-04T12:15:28.897 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-05-04T12:15:28.897 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid d9b985c4-0a0e-11ef-bc97-c7b262605968 -- ceph fs ls 2024-05-04T12:15:29.100 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/d9b985c4-0a0e-11ef-bc97-c7b262605968/mon.smithi079/config 2024-05-04T12:15:29.277 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:29 smithi146 ceph-mon[35311]: pgmap v192: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 6.8 KiB/s rd, 0 B/s wr, 6 op/s 2024-05-04T12:15:29.277 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:29 smithi146 ceph-mon[35311]: from='client.? 172.21.15.79:0/2280532527' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-05-04T12:15:29.277 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:29 smithi146 ceph-mon[35311]: from='client.? 172.21.15.79:0/3164138967' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-04T12:15:29.277 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:29 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:15:29.277 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:29 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:15:29.277 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:29 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-04T12:15:29.277 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:29 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-04T12:15:29.277 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:29 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:15:29.323 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:29 smithi079 ceph-mon[27222]: pgmap v192: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 6.8 KiB/s rd, 0 B/s wr, 6 op/s 2024-05-04T12:15:29.323 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:29 smithi079 ceph-mon[27222]: from='client.? 172.21.15.79:0/2280532527' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-05-04T12:15:29.323 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:29 smithi079 ceph-mon[27222]: from='client.? 172.21.15.79:0/3164138967' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-04T12:15:29.323 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:29 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:15:29.324 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:29 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:15:29.324 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:29 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-04T12:15:29.324 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:29 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-04T12:15:29.324 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:29 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:15:30.700 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:30.699+0000 7fba8717e700 1 -- 172.21.15.79:0/236840900 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fba80068580 msgr2=0x7fba800689e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:15:30.700 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:30.699+0000 7fba8717e700 1 --2- 172.21.15.79:0/236840900 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fba80068580 0x7fba800689e0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fba7000b0a0 tx=0x7fba7000b3b0 comp rx=0 tx=0).stop 2024-05-04T12:15:30.700 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:30.699+0000 7fba8717e700 1 -- 172.21.15.79:0/236840900 shutdown_connections 2024-05-04T12:15:30.700 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:30.699+0000 7fba8717e700 1 --2- 172.21.15.79:0/236840900 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fba80068580 0x7fba800689e0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:30.701 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:30.699+0000 7fba8717e700 1 --2- 172.21.15.79:0/236840900 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fba80106cd0 0x7fba801070b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:30.701 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:30.699+0000 7fba8717e700 1 -- 172.21.15.79:0/236840900 >> 172.21.15.79:0/236840900 conn(0x7fba80075670 msgr2=0x7fba80075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:15:30.701 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:30.699+0000 7fba8717e700 1 -- 172.21.15.79:0/236840900 shutdown_connections 2024-05-04T12:15:30.701 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:30.699+0000 7fba8717e700 1 -- 172.21.15.79:0/236840900 wait complete. 2024-05-04T12:15:30.701 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:30.699+0000 7fba8717e700 1 Processor -- start 2024-05-04T12:15:30.701 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:30.699+0000 7fba8717e700 1 -- start start 2024-05-04T12:15:30.701 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:30.700+0000 7fba8717e700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fba80068580 0x7fba80196980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:15:30.701 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:30.700+0000 7fba8717e700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fba80106cd0 0x7fba80196ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:15:30.701 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:30.700+0000 7fba8717e700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba80197550 con 0x7fba80068580 2024-05-04T12:15:30.701 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:30.700+0000 7fba8717e700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba8019b360 con 0x7fba80106cd0 2024-05-04T12:15:30.702 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:30.700+0000 7fba84f1a700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fba80068580 0x7fba80196980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:15:30.702 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:30.700+0000 7fba84f1a700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fba80068580 0x7fba80196980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:51454/0 (socket says 172.21.15.79:51454) 2024-05-04T12:15:30.702 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:30.700+0000 7fba84f1a700 1 -- 172.21.15.79:0/3716419003 learned_addr learned my addr 172.21.15.79:0/3716419003 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:15:30.702 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:30.700+0000 7fba7ffff700 1 --2- 172.21.15.79:0/3716419003 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fba80106cd0 0x7fba80196ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:15:30.702 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:30.700+0000 7fba84f1a700 1 -- 172.21.15.79:0/3716419003 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fba80106cd0 msgr2=0x7fba80196ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:15:30.702 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:30.700+0000 7fba84f1a700 1 --2- 172.21.15.79:0/3716419003 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fba80106cd0 0x7fba80196ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:30.702 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:30.700+0000 7fba84f1a700 1 -- 172.21.15.79:0/3716419003 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba70009d00 con 0x7fba80068580 2024-05-04T12:15:30.702 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:30.700+0000 7fba84f1a700 1 --2- 172.21.15.79:0/3716419003 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fba80068580 0x7fba80196980 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fba7800b4b0 tx=0x7fba7800b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:15:30.702 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:30.700+0000 7fba7dffb700 1 -- 172.21.15.79:0/3716419003 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fba78011800 con 0x7fba80068580 2024-05-04T12:15:30.702 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:30.700+0000 7fba7dffb700 1 -- 172.21.15.79:0/3716419003 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fba78011e10 con 0x7fba80068580 2024-05-04T12:15:30.702 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:30.700+0000 7fba8717e700 1 -- 172.21.15.79:0/3716419003 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba8019b640 con 0x7fba80068580 2024-05-04T12:15:30.703 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:30.700+0000 7fba7dffb700 1 -- 172.21.15.79:0/3716419003 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fba7800f2e0 con 0x7fba80068580 2024-05-04T12:15:30.703 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:30.701+0000 7fba8717e700 1 -- 172.21.15.79:0/3716419003 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba8019bb60 con 0x7fba80068580 2024-05-04T12:15:30.704 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:30.702+0000 7fba8717e700 1 -- 172.21.15.79:0/3716419003 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba801091d0 con 0x7fba80068580 2024-05-04T12:15:30.704 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:30.702+0000 7fba7dffb700 1 -- 172.21.15.79:0/3716419003 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7fba7800f510 con 0x7fba80068580 2024-05-04T12:15:30.704 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:30.702+0000 7fba7dffb700 1 --2- 172.21.15.79:0/3716419003 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7fba6c06c4e0 0x7fba6c06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:15:30.704 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:30.702+0000 7fba7dffb700 1 -- 172.21.15.79:0/3716419003 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fba7808b420 con 0x7fba80068580 2024-05-04T12:15:30.704 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:30.702+0000 7fba7ffff700 1 --2- 172.21.15.79:0/3716419003 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7fba6c06c4e0 0x7fba6c06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:15:30.704 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:30.703+0000 7fba7ffff700 1 --2- 172.21.15.79:0/3716419003 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7fba6c06c4e0 0x7fba6c06e9a0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fba70000f80 tx=0x7fba700060c0 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:15:30.706 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:30.707+0000 7fba7dffb700 1 -- 172.21.15.79:0/3716419003 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fba780596a0 con 0x7fba80068580 2024-05-04T12:15:30.938 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:30.939+0000 7fba8717e700 1 -- 172.21.15.79:0/3716419003 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fba8004f2e0 con 0x7fba80068580 2024-05-04T12:15:30.938 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:30.939+0000 7fba7dffb700 1 -- 172.21.15.79:0/3716419003 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v14) v1 ==== 53+0+83 (secure 0 0 0) 0x7fba78059230 con 0x7fba80068580 2024-05-04T12:15:30.939 INFO:teuthology.orchestra.run.smithi079.stdout:name: cephfs, metadata pool: cephfs.cephfs.meta, data pools: [cephfs.cephfs.data ] 2024-05-04T12:15:30.941 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:30.941+0000 7fba8717e700 1 -- 172.21.15.79:0/3716419003 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7fba6c06c4e0 msgr2=0x7fba6c06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:15:30.941 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:30.941+0000 7fba8717e700 1 --2- 172.21.15.79:0/3716419003 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7fba6c06c4e0 0x7fba6c06e9a0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fba70000f80 tx=0x7fba700060c0 comp rx=0 tx=0).stop 2024-05-04T12:15:30.941 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:30.941+0000 7fba8717e700 1 -- 172.21.15.79:0/3716419003 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fba80068580 msgr2=0x7fba80196980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:15:30.941 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:30.941+0000 7fba8717e700 1 --2- 172.21.15.79:0/3716419003 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fba80068580 0x7fba80196980 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fba7800b4b0 tx=0x7fba7800b7c0 comp rx=0 tx=0).stop 2024-05-04T12:15:30.941 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:30.941+0000 7fba8717e700 1 -- 172.21.15.79:0/3716419003 shutdown_connections 2024-05-04T12:15:30.941 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:30.941+0000 7fba8717e700 1 --2- 172.21.15.79:0/3716419003 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7fba6c06c4e0 0x7fba6c06e9a0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:30.941 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:30.941+0000 7fba8717e700 1 --2- 172.21.15.79:0/3716419003 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fba80068580 0x7fba80196980 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:30.942 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:30.941+0000 7fba8717e700 1 --2- 172.21.15.79:0/3716419003 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fba80106cd0 0x7fba80196ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:30.942 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:30.941+0000 7fba8717e700 1 -- 172.21.15.79:0/3716419003 >> 172.21.15.79:0/3716419003 conn(0x7fba80075670 msgr2=0x7fba800fede0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:15:30.942 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:30.941+0000 7fba8717e700 1 -- 172.21.15.79:0/3716419003 shutdown_connections 2024-05-04T12:15:30.942 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:30.941+0000 7fba8717e700 1 -- 172.21.15.79:0/3716419003 wait complete. 2024-05-04T12:15:31.195 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:31 smithi079 ceph-mon[27222]: pgmap v193: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 6.0 KiB/s rd, 0 B/s wr, 5 op/s 2024-05-04T12:15:31.195 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:31 smithi079 ceph-mon[27222]: from='client.? 172.21.15.79:0/3716419003' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-05-04T12:15:31.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:31 smithi146 ceph-mon[35311]: pgmap v193: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 6.0 KiB/s rd, 0 B/s wr, 5 op/s 2024-05-04T12:15:31.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:31 smithi146 ceph-mon[35311]: from='client.? 172.21.15.79:0/3716419003' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-05-04T12:15:31.601 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-05-04T12:15:31.601 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-05-04T12:15:31.601 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi079.front.sepia.ceph.com 2024-05-04T12:15:31.601 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-05-04T12:15:31.601 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-05-04T12:15:31.601 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-05-04T12:15:31.602 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-05-04T12:15:31.602 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-05-04T12:15:31.602 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-05-04T12:15:31.602 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-04T12:15:31.602 DEBUG:teuthology.orchestra.run.smithi079:> ip addr 2024-05-04T12:15:31.618 INFO:teuthology.orchestra.run.smithi079.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-05-04T12:15:31.619 INFO:teuthology.orchestra.run.smithi079.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-05-04T12:15:31.619 INFO:teuthology.orchestra.run.smithi079.stdout: inet 127.0.0.1/8 scope host lo 2024-05-04T12:15:31.619 INFO:teuthology.orchestra.run.smithi079.stdout: valid_lft forever preferred_lft forever 2024-05-04T12:15:31.619 INFO:teuthology.orchestra.run.smithi079.stdout: inet6 ::1/128 scope host 2024-05-04T12:15:31.619 INFO:teuthology.orchestra.run.smithi079.stdout: valid_lft forever preferred_lft forever 2024-05-04T12:15:31.619 INFO:teuthology.orchestra.run.smithi079.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-04T12:15:31.619 INFO:teuthology.orchestra.run.smithi079.stdout: link/ether 0c:c4:7a:d8:a1:6a brd ff:ff:ff:ff:ff:ff 2024-05-04T12:15:31.619 INFO:teuthology.orchestra.run.smithi079.stdout: altname enp7s0f0 2024-05-04T12:15:31.619 INFO:teuthology.orchestra.run.smithi079.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-04T12:15:31.619 INFO:teuthology.orchestra.run.smithi079.stdout: link/ether 0c:c4:7a:d8:a1:6b brd ff:ff:ff:ff:ff:ff 2024-05-04T12:15:31.619 INFO:teuthology.orchestra.run.smithi079.stdout: altname enp7s0f1 2024-05-04T12:15:31.619 INFO:teuthology.orchestra.run.smithi079.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-04T12:15:31.620 INFO:teuthology.orchestra.run.smithi079.stdout: link/ether 0c:c4:7a:8f:ce:86 brd ff:ff:ff:ff:ff:ff 2024-05-04T12:15:31.620 INFO:teuthology.orchestra.run.smithi079.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-05-04T12:15:31.620 INFO:teuthology.orchestra.run.smithi079.stdout: link/ether 0c:c4:7a:8f:ce:87 brd ff:ff:ff:ff:ff:ff 2024-05-04T12:15:31.620 INFO:teuthology.orchestra.run.smithi079.stdout: inet 172.21.15.79/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-05-04T12:15:31.620 INFO:teuthology.orchestra.run.smithi079.stdout: valid_lft 42234sec preferred_lft 42234sec 2024-05-04T12:15:31.620 INFO:teuthology.orchestra.run.smithi079.stdout: inet6 fe80::d9a9:5e05:f280:83dd/64 scope link noprefixroute 2024-05-04T12:15:31.620 INFO:teuthology.orchestra.run.smithi079.stdout: valid_lft forever preferred_lft forever 2024-05-04T12:15:31.620 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-05-04T12:15:31.620 DEBUG:teuthology.orchestra.run.smithi079:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-04T12:15:31.621 DEBUG:teuthology.orchestra.run.smithi079:> set -e 2024-05-04T12:15:31.621 DEBUG:teuthology.orchestra.run.smithi079:> sudo ip link add name ceph-brx type bridge 2024-05-04T12:15:31.621 DEBUG:teuthology.orchestra.run.smithi079:> sudo ip addr flush dev ceph-brx 2024-05-04T12:15:31.621 DEBUG:teuthology.orchestra.run.smithi079:> sudo ip link set ceph-brx up 2024-05-04T12:15:31.621 DEBUG:teuthology.orchestra.run.smithi079:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-05-04T12:15:31.621 DEBUG:teuthology.orchestra.run.smithi079:> ') 2024-05-04T12:15:31.708 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:31 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-04T12:15:31.801 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:31 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-04T12:15:31.808 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-04T12:15:31.808 DEBUG:teuthology.orchestra.run.smithi079:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-05-04T12:15:31.830 INFO:teuthology.orchestra.run.smithi079.stdout:1 2024-05-04T12:15:31.832 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-04T12:15:31.832 DEBUG:teuthology.orchestra.run.smithi079:> route 2024-05-04T12:15:31.910 INFO:teuthology.orchestra.run.smithi079.stdout:Kernel IP routing table 2024-05-04T12:15:31.910 INFO:teuthology.orchestra.run.smithi079.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-05-04T12:15:31.910 INFO:teuthology.orchestra.run.smithi079.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-05-04T12:15:31.911 INFO:teuthology.orchestra.run.smithi079.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-05-04T12:15:31.911 INFO:teuthology.orchestra.run.smithi079.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-05-04T12:15:31.911 DEBUG:teuthology.orchestra.run.smithi079:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-04T12:15:31.911 DEBUG:teuthology.orchestra.run.smithi079:> set -e 2024-05-04T12:15:31.911 DEBUG:teuthology.orchestra.run.smithi079:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-05-04T12:15:31.911 DEBUG:teuthology.orchestra.run.smithi079:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-05-04T12:15:31.911 DEBUG:teuthology.orchestra.run.smithi079:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-05-04T12:15:31.911 DEBUG:teuthology.orchestra.run.smithi079:> ') 2024-05-04T12:15:31.999 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:32 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-04T12:15:32.182 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:32 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-04T12:15:32.188 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-04T12:15:32.188 DEBUG:teuthology.orchestra.run.smithi079:> ip netns list 2024-05-04T12:15:32.203 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-04T12:15:32.204 DEBUG:teuthology.orchestra.run.smithi079:> ip netns list-id 2024-05-04T12:15:32.258 DEBUG:teuthology.orchestra.run.smithi079:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-04T12:15:32.258 DEBUG:teuthology.orchestra.run.smithi079:> set -e 2024-05-04T12:15:32.258 DEBUG:teuthology.orchestra.run.smithi079:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-04T12:15:32.258 DEBUG:teuthology.orchestra.run.smithi079:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-05-04T12:15:32.258 DEBUG:teuthology.orchestra.run.smithi079:> ') 2024-05-04T12:15:32.346 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:32 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-04T12:15:32.369 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:32 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-04T12:15:32.375 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-05-04T12:15:32.375 DEBUG:teuthology.orchestra.run.smithi079:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-04T12:15:32.376 DEBUG:teuthology.orchestra.run.smithi079:> set -e 2024-05-04T12:15:32.376 DEBUG:teuthology.orchestra.run.smithi079:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-05-04T12:15:32.376 DEBUG:teuthology.orchestra.run.smithi079:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-05-04T12:15:32.376 DEBUG:teuthology.orchestra.run.smithi079:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-05-04T12:15:32.376 DEBUG:teuthology.orchestra.run.smithi079:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-05-04T12:15:32.376 DEBUG:teuthology.orchestra.run.smithi079:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-05-04T12:15:32.376 DEBUG:teuthology.orchestra.run.smithi079:> ') 2024-05-04T12:15:32.424 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:32 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-04T12:15:32.508 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:32 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-04T12:15:32.516 DEBUG:teuthology.orchestra.run.smithi079:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-04T12:15:32.517 DEBUG:teuthology.orchestra.run.smithi079:> set -e 2024-05-04T12:15:32.517 DEBUG:teuthology.orchestra.run.smithi079:> sudo ip link set brx.0 up 2024-05-04T12:15:32.517 DEBUG:teuthology.orchestra.run.smithi079:> sudo ip link set dev brx.0 master ceph-brx 2024-05-04T12:15:32.517 DEBUG:teuthology.orchestra.run.smithi079:> ') 2024-05-04T12:15:32.564 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:32 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-04T12:15:32.584 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:32 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-04T12:15:32.592 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-04T12:15:32.592 DEBUG:teuthology.orchestra.run.smithi079:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-05-04T12:15:32.606 INFO:teuthology.orchestra.run.smithi079.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-05-04T12:15:32.607 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-04T12:15:32.607 DEBUG:teuthology.orchestra.run.smithi079:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-05-04T12:15:32.661 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-05-04T12:15:32.661 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-04T12:15:32.662 DEBUG:teuthology.orchestra.run.smithi079:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-05-04T12:15:32.751 INFO:teuthology.orchestra.run.smithi079.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-05-04T12:15:32.752 INFO:teuthology.orchestra.run.smithi079.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-05-04T12:15:32.752 INFO:teuthology.orchestra.run.smithi079.stdout:invalid new device string format 2024-05-04T12:15:32.851 INFO:teuthology.orchestra.run.smithi079.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-05-04T12:15:32.851 INFO:teuthology.orchestra.run.smithi079.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-05-04T12:15:32.851 INFO:teuthology.orchestra.run.smithi079.stdout:invalid new device string format 2024-05-04T12:15:32.851 INFO:teuthology.orchestra.run.smithi079.stdout:mount.ceph: resolved to: "172.21.15.79:3300,172.21.15.146:3300" 2024-05-04T12:15:32.851 INFO:teuthology.orchestra.run.smithi079.stdout:mount.ceph: switching to using v1 address with old syntax 2024-05-04T12:15:32.857 INFO:teuthology.orchestra.run.smithi079.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-05-04T12:15:32.857 INFO:teuthology.orchestra.run.smithi079.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-05-04T12:15:32.857 INFO:teuthology.orchestra.run.smithi079.stdout:invalid new device string format 2024-05-04T12:15:32.857 INFO:teuthology.orchestra.run.smithi079.stdout:mount.ceph: resolved to: "172.21.15.79:3300,172.21.15.146:3300" 2024-05-04T12:15:32.857 INFO:teuthology.orchestra.run.smithi079.stdout:mount.ceph: switching to using v1 address with old syntax 2024-05-04T12:15:32.857 INFO:teuthology.orchestra.run.smithi079.stdout:mount.ceph: resolved to: "172.21.15.79:6789,172.21.15.146:6789" 2024-05-04T12:15:32.857 INFO:teuthology.orchestra.run.smithi079.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.79:6789,172.21.15.146:6789:/ 2024-05-04T12:15:32.857 INFO:teuthology.orchestra.run.smithi079.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=d9b985c4-0a0e-11ef-bc97-c7b262605968" will pass to kernel 2024-05-04T12:15:32.858 INFO:teuthology.orchestra.run.smithi079.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-05-04T12:15:32.858 INFO:teuthology.orchestra.run.smithi079.stdout: You just mounted a file system that supports labels which does not 2024-05-04T12:15:32.859 INFO:teuthology.orchestra.run.smithi079.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-05-04T12:15:32.859 INFO:teuthology.orchestra.run.smithi079.stdout: applications will generate AVC messages and not be allowed access to 2024-05-04T12:15:32.859 INFO:teuthology.orchestra.run.smithi079.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-05-04T12:15:32.860 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-05-04T12:15:32.860 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-04T12:15:32.860 DEBUG:teuthology.orchestra.run.smithi079:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-05-04T12:15:32.883 DEBUG:teuthology.orchestra.run.smithi079:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-05-04T12:15:32.973 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:32 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-05-04T12:15:32.976 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:32 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-04T12:15:32.983 INFO:teuthology.orchestra.run.smithi079.stdout:client24317 2024-05-04T12:15:32.983 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-05-04T12:15:32.983 DEBUG:teuthology.orchestra.run.smithi079:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-04T12:15:32.999 DEBUG:teuthology.orchestra.run.smithi079:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-05-04T12:15:33.090 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:33 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-05-04T12:15:33.094 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:33 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-04T12:15:33.101 INFO:teuthology.orchestra.run.smithi079.stdout:client24317 2024-05-04T12:15:33.101 DEBUG:teuthology.orchestra.run.smithi079:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-05-04T12:15:33.151 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:33 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-05-04T12:15:33.156 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:33 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-04T12:15:33.162 INFO:teuthology.orchestra.run.smithi079.stdout:client24317 2024-05-04T12:15:33.162 DEBUG:teuthology.orchestra.run.smithi079:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/d9b985c4-0a0e-11ef-bc97-c7b262605968.client24317/status') 2024-05-04T12:15:33.227 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:33 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-04T12:15:33.238 INFO:teuthology.orchestra.run.smithi079.stdout:instance: client.24317 (3)172.21.15.79:0/72794610 2024-05-04T12:15:33.238 INFO:teuthology.orchestra.run.smithi079.stdout:blocklisted: false 2024-05-04T12:15:33.238 INFO:teuthology.orchestra.run.smithi079.stderr:0+1 records in 2024-05-04T12:15:33.239 INFO:teuthology.orchestra.run.smithi079.stderr:0+1 records out 2024-05-04T12:15:33.239 INFO:teuthology.orchestra.run.smithi079.stderr:69 bytes copied, 2.4999e-05 s, 2.8 MB/s 2024-05-04T12:15:33.239 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:33 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-04T12:15:33.246 INFO:tasks.kclient:client.1 config is {'syntax': 'v1'} 2024-05-04T12:15:33.246 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-05-04T12:15:33.246 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-05-04T12:15:33.246 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid d9b985c4-0a0e-11ef-bc97-c7b262605968 -- ceph fs ls 2024-05-04T12:15:33.276 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:33 smithi146 ceph-mon[35311]: pgmap v194: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 5.0 KiB/s rd, 0 B/s wr, 5 op/s 2024-05-04T12:15:33.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:33 smithi079 ceph-mon[27222]: pgmap v194: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 5.0 KiB/s rd, 0 B/s wr, 5 op/s 2024-05-04T12:15:33.450 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/d9b985c4-0a0e-11ef-bc97-c7b262605968/mon.smithi079/config 2024-05-04T12:15:34.823 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:34.823+0000 7ff1e7a0a700 1 -- 172.21.15.79:0/495180615 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff1e00fff50 msgr2=0x7ff1e0105840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:15:34.824 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:34.823+0000 7ff1e7a0a700 1 --2- 172.21.15.79:0/495180615 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff1e00fff50 0x7ff1e0105840 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7ff1d0009910 tx=0x7ff1d0009c20 comp rx=0 tx=0).stop 2024-05-04T12:15:34.824 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:34.823+0000 7ff1e7a0a700 1 -- 172.21.15.79:0/495180615 shutdown_connections 2024-05-04T12:15:34.824 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:34.823+0000 7ff1e7a0a700 1 --2- 172.21.15.79:0/495180615 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff1e00fff50 0x7ff1e0105840 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:34.824 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:34.823+0000 7ff1e7a0a700 1 --2- 172.21.15.79:0/495180615 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff1e00ff5a0 0x7ff1e00ff980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:34.824 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:34.823+0000 7ff1e7a0a700 1 -- 172.21.15.79:0/495180615 >> 172.21.15.79:0/495180615 conn(0x7ff1e0074e30 msgr2=0x7ff1e0075240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:15:34.824 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:34.823+0000 7ff1e7a0a700 1 -- 172.21.15.79:0/495180615 shutdown_connections 2024-05-04T12:15:34.824 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:34.823+0000 7ff1e7a0a700 1 -- 172.21.15.79:0/495180615 wait complete. 2024-05-04T12:15:34.824 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:34.824+0000 7ff1e7a0a700 1 Processor -- start 2024-05-04T12:15:34.824 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:34.824+0000 7ff1e7a0a700 1 -- start start 2024-05-04T12:15:34.824 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:34.824+0000 7ff1e7a0a700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff1e00ff5a0 0x7ff1e019c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:15:34.824 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:34.824+0000 7ff1e7a0a700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff1e00fff50 0x7ff1e019cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:15:34.824 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:34.824+0000 7ff1e7a0a700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1e019d260 con 0x7ff1e00ff5a0 2024-05-04T12:15:34.824 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:34.824+0000 7ff1e7a0a700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1e0196710 con 0x7ff1e00fff50 2024-05-04T12:15:34.824 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:34.825+0000 7ff1e57a6700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff1e00ff5a0 0x7ff1e019c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:15:34.825 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:34.825+0000 7ff1e57a6700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff1e00ff5a0 0x7ff1e019c690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:47722/0 (socket says 172.21.15.79:47722) 2024-05-04T12:15:34.825 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:34.825+0000 7ff1e57a6700 1 -- 172.21.15.79:0/1168742859 learned_addr learned my addr 172.21.15.79:0/1168742859 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:15:34.825 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:34.825+0000 7ff1e4fa5700 1 --2- 172.21.15.79:0/1168742859 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff1e00fff50 0x7ff1e019cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:15:34.825 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:34.825+0000 7ff1e57a6700 1 -- 172.21.15.79:0/1168742859 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff1e00fff50 msgr2=0x7ff1e019cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:15:34.825 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:34.825+0000 7ff1e57a6700 1 --2- 172.21.15.79:0/1168742859 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff1e00fff50 0x7ff1e019cbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:34.825 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:34.825+0000 7ff1e57a6700 1 -- 172.21.15.79:0/1168742859 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1d00095f0 con 0x7ff1e00ff5a0 2024-05-04T12:15:34.825 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:34.825+0000 7ff1e57a6700 1 --2- 172.21.15.79:0/1168742859 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff1e00ff5a0 0x7ff1e019c690 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7ff1d800ada0 tx=0x7ff1d800c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:15:34.825 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:34.825+0000 7ff1d67fc700 1 -- 172.21.15.79:0/1168742859 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff1d800f070 con 0x7ff1e00ff5a0 2024-05-04T12:15:34.825 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:34.825+0000 7ff1d67fc700 1 -- 172.21.15.79:0/1168742859 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7ff1d8007bf0 con 0x7ff1e00ff5a0 2024-05-04T12:15:34.825 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:34.825+0000 7ff1d67fc700 1 -- 172.21.15.79:0/1168742859 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff1d8005670 con 0x7ff1e00ff5a0 2024-05-04T12:15:34.825 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:34.825+0000 7ff1e7a0a700 1 -- 172.21.15.79:0/1168742859 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1e01969f0 con 0x7ff1e00ff5a0 2024-05-04T12:15:34.825 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:34.825+0000 7ff1e7a0a700 1 -- 172.21.15.79:0/1168742859 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1e0196f40 con 0x7ff1e00ff5a0 2024-05-04T12:15:34.826 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:34.826+0000 7ff1e7a0a700 1 -- 172.21.15.79:0/1168742859 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1e0109200 con 0x7ff1e00ff5a0 2024-05-04T12:15:34.826 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:34.827+0000 7ff1d67fc700 1 -- 172.21.15.79:0/1168742859 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7ff1d80058e0 con 0x7ff1e00ff5a0 2024-05-04T12:15:34.826 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:34.827+0000 7ff1d67fc700 1 --2- 172.21.15.79:0/1168742859 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7ff1cc06c4e0 0x7ff1cc06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:15:34.826 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:34.827+0000 7ff1d67fc700 1 -- 172.21.15.79:0/1168742859 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7ff1d808b2b0 con 0x7ff1e00ff5a0 2024-05-04T12:15:34.826 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:34.827+0000 7ff1e4fa5700 1 --2- 172.21.15.79:0/1168742859 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7ff1cc06c4e0 0x7ff1cc06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:15:34.826 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:34.828+0000 7ff1e4fa5700 1 --2- 172.21.15.79:0/1168742859 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7ff1cc06c4e0 0x7ff1cc06e9a0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7ff1e0197b80 tx=0x7ff1d001a040 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:15:34.830 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:34.832+0000 7ff1d67fc700 1 -- 172.21.15.79:0/1168742859 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7ff1d80594c0 con 0x7ff1e00ff5a0 2024-05-04T12:15:35.056 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:35.058+0000 7ff1e7a0a700 1 -- 172.21.15.79:0/1168742859 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7ff1e0197cc0 con 0x7ff1e00ff5a0 2024-05-04T12:15:35.057 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:35.058+0000 7ff1d67fc700 1 -- 172.21.15.79:0/1168742859 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v14) v1 ==== 53+0+83 (secure 0 0 0) 0x7ff1d8059050 con 0x7ff1e00ff5a0 2024-05-04T12:15:35.057 INFO:teuthology.orchestra.run.smithi079.stdout:name: cephfs, metadata pool: cephfs.cephfs.meta, data pools: [cephfs.cephfs.data ] 2024-05-04T12:15:35.059 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:35.060+0000 7ff1e7a0a700 1 -- 172.21.15.79:0/1168742859 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7ff1cc06c4e0 msgr2=0x7ff1cc06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:15:35.059 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:35.060+0000 7ff1e7a0a700 1 --2- 172.21.15.79:0/1168742859 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7ff1cc06c4e0 0x7ff1cc06e9a0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7ff1e0197b80 tx=0x7ff1d001a040 comp rx=0 tx=0).stop 2024-05-04T12:15:35.059 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:35.060+0000 7ff1e7a0a700 1 -- 172.21.15.79:0/1168742859 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff1e00ff5a0 msgr2=0x7ff1e019c690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:15:35.059 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:35.060+0000 7ff1e7a0a700 1 --2- 172.21.15.79:0/1168742859 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff1e00ff5a0 0x7ff1e019c690 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7ff1d800ada0 tx=0x7ff1d800c330 comp rx=0 tx=0).stop 2024-05-04T12:15:35.059 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:35.060+0000 7ff1e7a0a700 1 -- 172.21.15.79:0/1168742859 shutdown_connections 2024-05-04T12:15:35.059 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:35.060+0000 7ff1e7a0a700 1 --2- 172.21.15.79:0/1168742859 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7ff1cc06c4e0 0x7ff1cc06e9a0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:35.059 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:35.060+0000 7ff1e7a0a700 1 --2- 172.21.15.79:0/1168742859 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff1e00ff5a0 0x7ff1e019c690 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:35.059 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:35.060+0000 7ff1e7a0a700 1 --2- 172.21.15.79:0/1168742859 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff1e00fff50 0x7ff1e019cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:35.059 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:35.060+0000 7ff1e7a0a700 1 -- 172.21.15.79:0/1168742859 >> 172.21.15.79:0/1168742859 conn(0x7ff1e0074e30 msgr2=0x7ff1e00fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:15:35.059 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:35.060+0000 7ff1e7a0a700 1 -- 172.21.15.79:0/1168742859 shutdown_connections 2024-05-04T12:15:35.059 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:35.060+0000 7ff1e7a0a700 1 -- 172.21.15.79:0/1168742859 wait complete. 2024-05-04T12:15:35.195 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:35 smithi079 ceph-mon[27222]: pgmap v195: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 3.0 KiB/s rd, 0 B/s wr, 3 op/s 2024-05-04T12:15:35.195 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:35 smithi079 ceph-mon[27222]: from='client.? 172.21.15.79:0/1168742859' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-05-04T12:15:35.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:35 smithi146 ceph-mon[35311]: pgmap v195: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 3.0 KiB/s rd, 0 B/s wr, 3 op/s 2024-05-04T12:15:35.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:35 smithi146 ceph-mon[35311]: from='client.? 172.21.15.79:0/1168742859' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-05-04T12:15:35.700 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-05-04T12:15:35.700 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-05-04T12:15:35.700 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi146.front.sepia.ceph.com 2024-05-04T12:15:35.700 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-05-04T12:15:35.701 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-05-04T12:15:35.701 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-05-04T12:15:35.701 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-05-04T12:15:35.701 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-05-04T12:15:35.701 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-05-04T12:15:35.701 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-04T12:15:35.701 DEBUG:teuthology.orchestra.run.smithi146:> ip addr 2024-05-04T12:15:35.721 INFO:teuthology.orchestra.run.smithi146.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-05-04T12:15:35.721 INFO:teuthology.orchestra.run.smithi146.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-05-04T12:15:35.721 INFO:teuthology.orchestra.run.smithi146.stdout: inet 127.0.0.1/8 scope host lo 2024-05-04T12:15:35.721 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-05-04T12:15:35.721 INFO:teuthology.orchestra.run.smithi146.stdout: inet6 ::1/128 scope host 2024-05-04T12:15:35.721 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-05-04T12:15:35.721 INFO:teuthology.orchestra.run.smithi146.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-04T12:15:35.721 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 0c:c4:7a:d9:42:6e brd ff:ff:ff:ff:ff:ff 2024-05-04T12:15:35.721 INFO:teuthology.orchestra.run.smithi146.stdout: altname enp7s0f0 2024-05-04T12:15:35.721 INFO:teuthology.orchestra.run.smithi146.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-04T12:15:35.721 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 0c:c4:7a:d9:42:6f brd ff:ff:ff:ff:ff:ff 2024-05-04T12:15:35.721 INFO:teuthology.orchestra.run.smithi146.stdout: altname enp7s0f1 2024-05-04T12:15:35.721 INFO:teuthology.orchestra.run.smithi146.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-04T12:15:35.722 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 0c:c4:7a:88:76:fc brd ff:ff:ff:ff:ff:ff 2024-05-04T12:15:35.722 INFO:teuthology.orchestra.run.smithi146.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-05-04T12:15:35.722 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 0c:c4:7a:88:76:fd brd ff:ff:ff:ff:ff:ff 2024-05-04T12:15:35.722 INFO:teuthology.orchestra.run.smithi146.stdout: inet 172.21.15.146/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-05-04T12:15:35.722 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft 42233sec preferred_lft 42233sec 2024-05-04T12:15:35.722 INFO:teuthology.orchestra.run.smithi146.stdout: inet6 fe80::8659:ede5:f087:6111/64 scope link noprefixroute 2024-05-04T12:15:35.722 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-05-04T12:15:35.722 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-05-04T12:15:35.722 DEBUG:teuthology.orchestra.run.smithi146:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-04T12:15:35.722 DEBUG:teuthology.orchestra.run.smithi146:> set -e 2024-05-04T12:15:35.722 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip link add name ceph-brx type bridge 2024-05-04T12:15:35.722 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip addr flush dev ceph-brx 2024-05-04T12:15:35.723 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip link set ceph-brx up 2024-05-04T12:15:35.723 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-05-04T12:15:35.723 DEBUG:teuthology.orchestra.run.smithi146:> ') 2024-05-04T12:15:35.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:15:35 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-04T12:15:35.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:15:35 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-04T12:15:35.913 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-04T12:15:35.913 DEBUG:teuthology.orchestra.run.smithi146:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-05-04T12:15:35.935 INFO:teuthology.orchestra.run.smithi146.stdout:1 2024-05-04T12:15:35.937 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-04T12:15:35.937 DEBUG:teuthology.orchestra.run.smithi146:> route 2024-05-04T12:15:36.003 INFO:teuthology.orchestra.run.smithi146.stdout:Kernel IP routing table 2024-05-04T12:15:36.003 INFO:teuthology.orchestra.run.smithi146.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-05-04T12:15:36.003 INFO:teuthology.orchestra.run.smithi146.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-05-04T12:15:36.003 INFO:teuthology.orchestra.run.smithi146.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-05-04T12:15:36.003 INFO:teuthology.orchestra.run.smithi146.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-05-04T12:15:36.003 DEBUG:teuthology.orchestra.run.smithi146:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-04T12:15:36.003 DEBUG:teuthology.orchestra.run.smithi146:> set -e 2024-05-04T12:15:36.003 DEBUG:teuthology.orchestra.run.smithi146:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-05-04T12:15:36.003 DEBUG:teuthology.orchestra.run.smithi146:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-05-04T12:15:36.003 DEBUG:teuthology.orchestra.run.smithi146:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-05-04T12:15:36.003 DEBUG:teuthology.orchestra.run.smithi146:> ') 2024-05-04T12:15:36.091 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:15:36 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-04T12:15:36.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:15:36 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-04T12:15:36.270 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-04T12:15:36.270 DEBUG:teuthology.orchestra.run.smithi146:> ip netns list 2024-05-04T12:15:36.285 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-04T12:15:36.285 DEBUG:teuthology.orchestra.run.smithi146:> ip netns list-id 2024-05-04T12:15:36.341 DEBUG:teuthology.orchestra.run.smithi146:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-04T12:15:36.341 DEBUG:teuthology.orchestra.run.smithi146:> set -e 2024-05-04T12:15:36.341 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-05-04T12:15:36.341 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-05-04T12:15:36.341 DEBUG:teuthology.orchestra.run.smithi146:> ') 2024-05-04T12:15:36.430 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:15:36 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-04T12:15:36.453 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:15:36 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-04T12:15:36.459 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-05-04T12:15:36.460 DEBUG:teuthology.orchestra.run.smithi146:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-04T12:15:36.460 DEBUG:teuthology.orchestra.run.smithi146:> set -e 2024-05-04T12:15:36.460 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-05-04T12:15:36.460 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-05-04T12:15:36.460 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-05-04T12:15:36.460 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-05-04T12:15:36.460 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-05-04T12:15:36.460 DEBUG:teuthology.orchestra.run.smithi146:> ') 2024-05-04T12:15:36.509 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:15:36 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-04T12:15:36.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:15:36 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-04T12:15:36.594 DEBUG:teuthology.orchestra.run.smithi146:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-04T12:15:36.594 DEBUG:teuthology.orchestra.run.smithi146:> set -e 2024-05-04T12:15:36.594 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip link set brx.0 up 2024-05-04T12:15:36.595 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip link set dev brx.0 master ceph-brx 2024-05-04T12:15:36.595 DEBUG:teuthology.orchestra.run.smithi146:> ') 2024-05-04T12:15:36.642 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:15:36 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-04T12:15:36.664 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:15:36 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-04T12:15:36.671 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-04T12:15:36.671 DEBUG:teuthology.orchestra.run.smithi146:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-05-04T12:15:36.686 INFO:teuthology.orchestra.run.smithi146.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-05-04T12:15:36.686 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-04T12:15:36.686 DEBUG:teuthology.orchestra.run.smithi146:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-05-04T12:15:36.741 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-05-04T12:15:36.741 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-04T12:15:36.741 DEBUG:teuthology.orchestra.run.smithi146:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-05-04T12:15:36.854 INFO:teuthology.orchestra.run.smithi146.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-05-04T12:15:36.854 INFO:teuthology.orchestra.run.smithi146.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-05-04T12:15:36.854 INFO:teuthology.orchestra.run.smithi146.stdout:invalid new device string format 2024-05-04T12:15:36.951 INFO:teuthology.orchestra.run.smithi146.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-05-04T12:15:36.951 INFO:teuthology.orchestra.run.smithi146.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-05-04T12:15:36.951 INFO:teuthology.orchestra.run.smithi146.stdout:invalid new device string format 2024-05-04T12:15:36.951 INFO:teuthology.orchestra.run.smithi146.stdout:mount.ceph: resolved to: "172.21.15.79:3300,172.21.15.146:3300" 2024-05-04T12:15:36.952 INFO:teuthology.orchestra.run.smithi146.stdout:mount.ceph: switching to using v1 address with old syntax 2024-05-04T12:15:36.960 INFO:teuthology.orchestra.run.smithi146.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-05-04T12:15:36.961 INFO:teuthology.orchestra.run.smithi146.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-05-04T12:15:36.961 INFO:teuthology.orchestra.run.smithi146.stdout:invalid new device string format 2024-05-04T12:15:36.961 INFO:teuthology.orchestra.run.smithi146.stdout:mount.ceph: resolved to: "172.21.15.79:3300,172.21.15.146:3300" 2024-05-04T12:15:36.961 INFO:teuthology.orchestra.run.smithi146.stdout:mount.ceph: switching to using v1 address with old syntax 2024-05-04T12:15:36.961 INFO:teuthology.orchestra.run.smithi146.stdout:mount.ceph: resolved to: "172.21.15.79:6789,172.21.15.146:6789" 2024-05-04T12:15:36.961 INFO:teuthology.orchestra.run.smithi146.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.79:6789,172.21.15.146:6789:/ 2024-05-04T12:15:36.961 INFO:teuthology.orchestra.run.smithi146.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs,key=1,fsid=d9b985c4-0a0e-11ef-bc97-c7b262605968" will pass to kernel 2024-05-04T12:15:36.961 INFO:teuthology.orchestra.run.smithi146.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2024-05-04T12:15:36.961 INFO:teuthology.orchestra.run.smithi146.stdout: You just mounted a file system that supports labels which does not 2024-05-04T12:15:36.962 INFO:teuthology.orchestra.run.smithi146.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-05-04T12:15:36.962 INFO:teuthology.orchestra.run.smithi146.stdout: applications will generate AVC messages and not be allowed access to 2024-05-04T12:15:36.962 INFO:teuthology.orchestra.run.smithi146.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-05-04T12:15:36.963 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-05-04T12:15:36.963 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-04T12:15:36.964 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-05-04T12:15:36.987 DEBUG:teuthology.orchestra.run.smithi146:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-05-04T12:15:37.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:15:37 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-05-04T12:15:37.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:15:37 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-04T12:15:37.084 INFO:teuthology.orchestra.run.smithi146.stdout:client14534 2024-05-04T12:15:37.084 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-05-04T12:15:37.085 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-04T12:15:37.101 DEBUG:teuthology.orchestra.run.smithi146:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-05-04T12:15:37.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:15:37 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-05-04T12:15:37.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:15:37 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-04T12:15:37.196 INFO:teuthology.orchestra.run.smithi146.stdout:client14534 2024-05-04T12:15:37.196 DEBUG:teuthology.orchestra.run.smithi146:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-05-04T12:15:37.244 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:15:37 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-05-04T12:15:37.245 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:15:37 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-04T12:15:37.251 INFO:teuthology.orchestra.run.smithi146.stdout:client14534 2024-05-04T12:15:37.252 DEBUG:teuthology.orchestra.run.smithi146:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/d9b985c4-0a0e-11ef-bc97-c7b262605968.client14534/status') 2024-05-04T12:15:37.276 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:37 smithi146 ceph-mon[35311]: pgmap v196: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-04T12:15:37.299 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:15:37 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-04T12:15:37.309 INFO:teuthology.orchestra.run.smithi146.stdout:instance: client.14534 (3)172.21.15.146:0/1630758539 2024-05-04T12:15:37.309 INFO:teuthology.orchestra.run.smithi146.stdout:blocklisted: false 2024-05-04T12:15:37.309 INFO:teuthology.orchestra.run.smithi146.stderr:0+1 records in 2024-05-04T12:15:37.309 INFO:teuthology.orchestra.run.smithi146.stderr:0+1 records out 2024-05-04T12:15:37.309 INFO:teuthology.orchestra.run.smithi146.stderr:72 bytes copied, 2.366e-05 s, 3.0 MB/s 2024-05-04T12:15:37.310 INFO:teuthology.orchestra.run.smithi146.stderr:2024-05-04T12:15:37 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-04T12:15:37.316 INFO:teuthology.run_tasks:Running task print... 2024-05-04T12:15:37.326 INFO:teuthology.task.print:**** done client 2024-05-04T12:15:37.326 INFO:teuthology.run_tasks:Running task parallel... 2024-05-04T12:15:37.347 INFO:teuthology.task.parallel:starting parallel... 2024-05-04T12:15:37.347 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-05-04T12:15:37.348 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-05-04T12:15:37.348 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi079.front.sepia.ceph.com 2024-05-04T12:15:37.348 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d9b985c4-0a0e-11ef-bc97-c7b262605968 -e sha1=09dbd6bba52d19ba471c33bc15c008e6ad158ea6 -- bash -c 'ceph config set mgr mgr/orchestrator/fail_fs false || true' 2024-05-04T12:15:37.348 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-05-04T12:15:37.348 INFO:teuthology.task.sequential:In sequential, running task workunit... 2024-05-04T12:15:37.352 INFO:tasks.workunit:Pulling workunits from ref 09dbd6bba52d19ba471c33bc15c008e6ad158ea6 2024-05-04T12:15:37.352 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-05-04T12:15:37.352 INFO:tasks.workunit:timeout=3h 2024-05-04T12:15:37.352 INFO:tasks.workunit:cleanup=True 2024-05-04T12:15:37.352 DEBUG:teuthology.orchestra.run.smithi079:> stat -- /home/ubuntu/cephtest/mnt.0 2024-05-04T12:15:37.372 INFO:teuthology.orchestra.run.smithi079.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-05-04T12:15:37.373 INFO:teuthology.orchestra.run.smithi079.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-05-04T12:15:37.373 INFO:teuthology.orchestra.run.smithi079.stdout:Device: 91h/145d Inode: 1 Links: 2 2024-05-04T12:15:37.373 INFO:teuthology.orchestra.run.smithi079.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-04T12:15:37.373 INFO:teuthology.orchestra.run.smithi079.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-05-04T12:15:37.373 INFO:teuthology.orchestra.run.smithi079.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2024-05-04T12:15:37.373 INFO:teuthology.orchestra.run.smithi079.stdout:Modify: 2024-05-04 12:15:05.100831728 +0000 2024-05-04T12:15:37.373 INFO:teuthology.orchestra.run.smithi079.stdout:Change: 2024-05-04 12:15:32.883056130 +0000 2024-05-04T12:15:37.373 INFO:teuthology.orchestra.run.smithi079.stdout: Birth: 2024-05-04 12:15:05.100831728 +0000 2024-05-04T12:15:37.373 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-05-04T12:15:37.373 DEBUG:teuthology.orchestra.run.smithi079:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-05-04T12:15:37.379 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:37 smithi079 ceph-mon[27222]: pgmap v196: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-04T12:15:37.409 DEBUG:teuthology.orchestra.run.smithi146:> stat -- /home/ubuntu/cephtest/mnt.1 2024-05-04T12:15:37.425 INFO:teuthology.orchestra.run.smithi146.stdout: File: /home/ubuntu/cephtest/mnt.1 2024-05-04T12:15:37.425 INFO:teuthology.orchestra.run.smithi146.stdout: Size: 1 Blocks: 0 IO Block: 65536 directory 2024-05-04T12:15:37.425 INFO:teuthology.orchestra.run.smithi146.stdout:Device: 7bh/123d Inode: 1 Links: 3 2024-05-04T12:15:37.426 INFO:teuthology.orchestra.run.smithi146.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-04T12:15:37.426 INFO:teuthology.orchestra.run.smithi146.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-05-04T12:15:37.426 INFO:teuthology.orchestra.run.smithi146.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2024-05-04T12:15:37.426 INFO:teuthology.orchestra.run.smithi146.stdout:Modify: 2024-05-04 12:15:37.405954929 +0000 2024-05-04T12:15:37.426 INFO:teuthology.orchestra.run.smithi146.stdout:Change: 2024-05-04 12:15:37.405954929 +0000 2024-05-04T12:15:37.426 INFO:teuthology.orchestra.run.smithi146.stdout: Birth: 2024-05-04 12:15:05.100831728 +0000 2024-05-04T12:15:37.426 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.1 2024-05-04T12:15:37.426 DEBUG:teuthology.orchestra.run.smithi146:> cd -- /home/ubuntu/cephtest/mnt.1 && sudo install -d -m 0755 --owner=ubuntu -- client.1 2024-05-04T12:15:37.492 DEBUG:teuthology.orchestra.run.smithi079:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 09dbd6bba52d19ba471c33bc15c008e6ad158ea6 2024-05-04T12:15:37.493 DEBUG:teuthology.orchestra.run.smithi146:> rm -rf /home/ubuntu/cephtest/clone.client.1 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.1 && cd /home/ubuntu/cephtest/clone.client.1 && git checkout 09dbd6bba52d19ba471c33bc15c008e6ad158ea6 2024-05-04T12:15:37.509 INFO:tasks.workunit.client.0.smithi079.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-05-04T12:15:37.547 INFO:tasks.workunit.client.1.smithi146.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.1'... 2024-05-04T12:15:37.577 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/d9b985c4-0a0e-11ef-bc97-c7b262605968/mon.smithi079/config 2024-05-04T12:15:39.179 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:39.178+0000 7f69a9a09700 1 -- 172.21.15.79:0/2541701233 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f69a40ffba0 msgr2=0x7f69a40fffc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:15:39.179 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:39.178+0000 7f69a9a09700 1 --2- 172.21.15.79:0/2541701233 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f69a40ffba0 0x7f69a40fffc0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f698c009910 tx=0x7f698c009c20 comp rx=0 tx=0).stop 2024-05-04T12:15:39.180 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:39.178+0000 7f69a9a09700 1 -- 172.21.15.79:0/2541701233 shutdown_connections 2024-05-04T12:15:39.180 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:39.178+0000 7f69a9a09700 1 --2- 172.21.15.79:0/2541701233 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f69a4100da0 0x7f69a4101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:39.180 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:39.178+0000 7f69a9a09700 1 --2- 172.21.15.79:0/2541701233 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f69a40ffba0 0x7f69a40fffc0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:39.180 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:39.178+0000 7f69a9a09700 1 -- 172.21.15.79:0/2541701233 >> 172.21.15.79:0/2541701233 conn(0x7f69a40fb120 msgr2=0x7f69a40fd580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:15:39.180 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:39.178+0000 7f69a9a09700 1 -- 172.21.15.79:0/2541701233 shutdown_connections 2024-05-04T12:15:39.180 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:39.178+0000 7f69a9a09700 1 -- 172.21.15.79:0/2541701233 wait complete. 2024-05-04T12:15:39.180 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:39.179+0000 7f69a9a09700 1 Processor -- start 2024-05-04T12:15:39.180 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:39.179+0000 7f69a9a09700 1 -- start start 2024-05-04T12:15:39.180 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:39.179+0000 7f69a9a09700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f69a40ffba0 0x7f69a4196530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:15:39.180 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:39.179+0000 7f69a9a09700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f69a4100da0 0x7f69a4196a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:15:39.180 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:39.179+0000 7f69a9a09700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69a4197090 con 0x7f69a40ffba0 2024-05-04T12:15:39.180 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:39.179+0000 7f69a9a09700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69a41971d0 con 0x7f69a4100da0 2024-05-04T12:15:39.180 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:39.179+0000 7f69a2ffd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f69a40ffba0 0x7f69a4196530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:15:39.180 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:39.179+0000 7f69a2ffd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f69a40ffba0 0x7f69a4196530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:47754/0 (socket says 172.21.15.79:47754) 2024-05-04T12:15:39.181 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:39.179+0000 7f69a2ffd700 1 -- 172.21.15.79:0/2224601540 learned_addr learned my addr 172.21.15.79:0/2224601540 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:15:39.181 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:39.180+0000 7f69a27fc700 1 --2- 172.21.15.79:0/2224601540 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f69a4100da0 0x7f69a4196a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:15:39.181 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:39.180+0000 7f69a2ffd700 1 -- 172.21.15.79:0/2224601540 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f69a4100da0 msgr2=0x7f69a4196a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:15:39.181 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:39.180+0000 7f69a2ffd700 1 --2- 172.21.15.79:0/2224601540 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f69a4100da0 0x7f69a4196a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:39.181 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:39.180+0000 7f69a2ffd700 1 -- 172.21.15.79:0/2224601540 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f698c0095f0 con 0x7f69a40ffba0 2024-05-04T12:15:39.181 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:39.180+0000 7f69a2ffd700 1 --2- 172.21.15.79:0/2224601540 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f69a40ffba0 0x7f69a4196530 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f698c00bda0 tx=0x7f698c00bdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:15:39.181 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:39.180+0000 7f69a8a07700 1 -- 172.21.15.79:0/2224601540 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f698c01d070 con 0x7f69a40ffba0 2024-05-04T12:15:39.181 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:39.180+0000 7f69a8a07700 1 -- 172.21.15.79:0/2224601540 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f698c022470 con 0x7f69a40ffba0 2024-05-04T12:15:39.181 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:39.180+0000 7f69a9a09700 1 -- 172.21.15.79:0/2224601540 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69a419bc20 con 0x7f69a40ffba0 2024-05-04T12:15:39.181 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:39.180+0000 7f69a8a07700 1 -- 172.21.15.79:0/2224601540 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f698c00f650 con 0x7f69a40ffba0 2024-05-04T12:15:39.181 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:39.180+0000 7f69a9a09700 1 -- 172.21.15.79:0/2224601540 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69a419c110 con 0x7f69a40ffba0 2024-05-04T12:15:39.182 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:39.181+0000 7f69a9a09700 1 -- 172.21.15.79:0/2224601540 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69a4108780 con 0x7f69a40ffba0 2024-05-04T12:15:39.182 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:39.182+0000 7f69a8a07700 1 -- 172.21.15.79:0/2224601540 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f698c004af0 con 0x7f69a40ffba0 2024-05-04T12:15:39.182 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:39.182+0000 7f69a8a07700 1 --2- 172.21.15.79:0/2224601540 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f699006c4e0 0x7f699006e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:15:39.182 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:39.182+0000 7f69a27fc700 1 --2- 172.21.15.79:0/2224601540 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f699006c4e0 0x7f699006e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:15:39.182 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:39.182+0000 7f69a8a07700 1 -- 172.21.15.79:0/2224601540 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f698c08cf00 con 0x7f69a40ffba0 2024-05-04T12:15:39.182 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:39.183+0000 7f69a27fc700 1 --2- 172.21.15.79:0/2224601540 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f699006c4e0 0x7f699006e9a0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f6994009a10 tx=0x7f6994009190 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:15:39.186 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:39.187+0000 7f69a8a07700 1 -- 172.21.15.79:0/2224601540 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f698c05b1c0 con 0x7f69a40ffba0 2024-05-04T12:15:39.195 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:39 smithi079 ceph-mon[27222]: pgmap v197: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 255 B/s wr, 0 op/s 2024-05-04T12:15:39.375 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:39.375+0000 7f69a9a09700 1 -- 172.21.15.79:0/2224601540 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command([{prefix=config set, name=mgr/orchestrator/fail_fs}] v 0) v1 -- 0x7f69a419c810 con 0x7f69a40ffba0 2024-05-04T12:15:39.381 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:39.382+0000 7f69a8a07700 1 -- 172.21.15.79:0/2224601540 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/orchestrator/fail_fs}]=0 v22)=0 v22) v1 ==== 126+0+0 (secure 0 0 0) 0x7f698c05ad50 con 0x7f69a40ffba0 2024-05-04T12:15:39.384 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:39.384+0000 7f69a9a09700 1 -- 172.21.15.79:0/2224601540 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f699006c4e0 msgr2=0x7f699006e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:15:39.384 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:39.384+0000 7f69a9a09700 1 --2- 172.21.15.79:0/2224601540 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f699006c4e0 0x7f699006e9a0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f6994009a10 tx=0x7f6994009190 comp rx=0 tx=0).stop 2024-05-04T12:15:39.384 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:39.384+0000 7f69a9a09700 1 -- 172.21.15.79:0/2224601540 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f69a40ffba0 msgr2=0x7f69a4196530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:15:39.384 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:39.384+0000 7f69a9a09700 1 --2- 172.21.15.79:0/2224601540 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f69a40ffba0 0x7f69a4196530 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f698c00bda0 tx=0x7f698c00bdd0 comp rx=0 tx=0).stop 2024-05-04T12:15:39.384 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:39.384+0000 7f69a9a09700 1 -- 172.21.15.79:0/2224601540 shutdown_connections 2024-05-04T12:15:39.384 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:39.384+0000 7f69a9a09700 1 --2- 172.21.15.79:0/2224601540 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f699006c4e0 0x7f699006e9a0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:39.384 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:39.384+0000 7f69a9a09700 1 --2- 172.21.15.79:0/2224601540 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f69a40ffba0 0x7f69a4196530 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:39.384 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:39.384+0000 7f69a9a09700 1 --2- 172.21.15.79:0/2224601540 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f69a4100da0 0x7f69a4196a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:39.384 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:39.384+0000 7f69a9a09700 1 -- 172.21.15.79:0/2224601540 >> 172.21.15.79:0/2224601540 conn(0x7f69a40fb120 msgr2=0x7f69a4068e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:15:39.384 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:39.384+0000 7f69a9a09700 1 -- 172.21.15.79:0/2224601540 shutdown_connections 2024-05-04T12:15:39.384 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:39.384+0000 7f69a9a09700 1 -- 172.21.15.79:0/2224601540 wait complete. 2024-05-04T12:15:39.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:39 smithi146 ceph-mon[35311]: pgmap v197: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 255 B/s wr, 0 op/s 2024-05-04T12:15:40.024 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-05-04T12:15:40.025 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi079.front.sepia.ceph.com 2024-05-04T12:15:40.025 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d9b985c4-0a0e-11ef-bc97-c7b262605968 -e sha1=09dbd6bba52d19ba471c33bc15c008e6ad158ea6 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2024-05-04T12:15:40.272 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/d9b985c4-0a0e-11ef-bc97-c7b262605968/mon.smithi079/config 2024-05-04T12:15:40.695 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:40 smithi079 ceph-mon[27222]: from='client.? 172.21.15.79:0/2224601540' entity='client.admin' 2024-05-04T12:15:40.695 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:40 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-04T12:15:40.695 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:40 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-04T12:15:40.696 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:40 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-04T12:15:40.696 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:40 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:15:40.696 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:40 smithi079 ceph-mon[27222]: pgmap v198: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 255 B/s wr, 0 op/s 2024-05-04T12:15:40.777 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:40 smithi146 ceph-mon[35311]: from='client.? 172.21.15.79:0/2224601540' entity='client.admin' 2024-05-04T12:15:40.777 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:40 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-04T12:15:40.777 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:40 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-04T12:15:40.777 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:40 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-04T12:15:40.777 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:40 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:15:40.777 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:40 smithi146 ceph-mon[35311]: pgmap v198: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 255 B/s wr, 0 op/s 2024-05-04T12:15:41.938 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:41.932+0000 7faf71361700 1 -- 172.21.15.79:0/1068917633 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7faf6c0ffba0 msgr2=0x7faf6c0fffc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:15:41.939 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:41.932+0000 7faf71361700 1 --2- 172.21.15.79:0/1068917633 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7faf6c0ffba0 0x7faf6c0fffc0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7faf5c009910 tx=0x7faf5c009c20 comp rx=0 tx=0).stop 2024-05-04T12:15:41.939 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:41.932+0000 7faf71361700 1 -- 172.21.15.79:0/1068917633 shutdown_connections 2024-05-04T12:15:41.939 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:41.932+0000 7faf71361700 1 --2- 172.21.15.79:0/1068917633 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faf6c100da0 0x7faf6c101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:41.939 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:41.932+0000 7faf71361700 1 --2- 172.21.15.79:0/1068917633 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7faf6c0ffba0 0x7faf6c0fffc0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:41.939 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:41.932+0000 7faf71361700 1 -- 172.21.15.79:0/1068917633 >> 172.21.15.79:0/1068917633 conn(0x7faf6c0fb120 msgr2=0x7faf6c0fd580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:15:41.939 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:41.932+0000 7faf71361700 1 -- 172.21.15.79:0/1068917633 shutdown_connections 2024-05-04T12:15:41.939 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:41.932+0000 7faf71361700 1 -- 172.21.15.79:0/1068917633 wait complete. 2024-05-04T12:15:41.939 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:41.933+0000 7faf71361700 1 Processor -- start 2024-05-04T12:15:41.939 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:41.933+0000 7faf71361700 1 -- start start 2024-05-04T12:15:41.939 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:41.933+0000 7faf71361700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faf6c0ffba0 0x7faf6c069990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:15:41.939 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:41.933+0000 7faf71361700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7faf6c100da0 0x7faf6c19b210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:15:41.939 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:41.933+0000 7faf71361700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf6c19b7e0 con 0x7faf6c100da0 2024-05-04T12:15:41.939 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:41.933+0000 7faf71361700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf6c19b950 con 0x7faf6c0ffba0 2024-05-04T12:15:41.939 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:41.933+0000 7faf6a7fc700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7faf6c100da0 0x7faf6c19b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:15:41.940 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:41.933+0000 7faf6a7fc700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7faf6c100da0 0x7faf6c19b210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:47762/0 (socket says 172.21.15.79:47762) 2024-05-04T12:15:41.940 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:41.933+0000 7faf6a7fc700 1 -- 172.21.15.79:0/1561890553 learned_addr learned my addr 172.21.15.79:0/1561890553 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:15:41.940 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:41.933+0000 7faf6affd700 1 --2- 172.21.15.79:0/1561890553 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faf6c0ffba0 0x7faf6c069990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:15:41.940 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:41.934+0000 7faf6a7fc700 1 -- 172.21.15.79:0/1561890553 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faf6c0ffba0 msgr2=0x7faf6c069990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:15:41.940 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:41.934+0000 7faf6a7fc700 1 --2- 172.21.15.79:0/1561890553 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faf6c0ffba0 0x7faf6c069990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:41.940 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:41.934+0000 7faf6a7fc700 1 -- 172.21.15.79:0/1561890553 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf5c0095f0 con 0x7faf6c100da0 2024-05-04T12:15:41.940 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:41.934+0000 7faf6a7fc700 1 --2- 172.21.15.79:0/1561890553 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7faf6c100da0 0x7faf6c19b210 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7faf6000e920 tx=0x7faf6000ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:15:41.940 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:41.934+0000 7faf53fff700 1 -- 172.21.15.79:0/1561890553 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faf6000c9d0 con 0x7faf6c100da0 2024-05-04T12:15:41.940 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:41.934+0000 7faf53fff700 1 -- 172.21.15.79:0/1561890553 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7faf60010430 con 0x7faf6c100da0 2024-05-04T12:15:41.940 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:41.934+0000 7faf71361700 1 -- 172.21.15.79:0/1561890553 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf6c19bc30 con 0x7faf6c100da0 2024-05-04T12:15:41.940 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:41.934+0000 7faf53fff700 1 -- 172.21.15.79:0/1561890553 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faf60018650 con 0x7faf6c100da0 2024-05-04T12:15:41.940 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:41.934+0000 7faf71361700 1 -- 172.21.15.79:0/1561890553 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf6c19c180 con 0x7faf6c100da0 2024-05-04T12:15:41.940 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:41.935+0000 7faf71361700 1 -- 172.21.15.79:0/1561890553 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf6c108700 con 0x7faf6c100da0 2024-05-04T12:15:41.940 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:41.936+0000 7faf53fff700 1 -- 172.21.15.79:0/1561890553 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7faf600105a0 con 0x7faf6c100da0 2024-05-04T12:15:41.940 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:41.936+0000 7faf53fff700 1 --2- 172.21.15.79:0/1561890553 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7faf5406c4e0 0x7faf5406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:15:41.940 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:41.936+0000 7faf53fff700 1 -- 172.21.15.79:0/1561890553 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7faf60014070 con 0x7faf6c100da0 2024-05-04T12:15:41.941 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:41.936+0000 7faf6affd700 1 --2- 172.21.15.79:0/1561890553 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7faf5406c4e0 0x7faf5406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:15:41.941 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:41.937+0000 7faf6affd700 1 --2- 172.21.15.79:0/1561890553 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7faf5406c4e0 0x7faf5406e9a0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7faf5c00b6f0 tx=0x7faf5c005940 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:15:41.941 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:41.941+0000 7faf53fff700 1 -- 172.21.15.79:0/1561890553 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7faf600520e0 con 0x7faf6c100da0 2024-05-04T12:15:42.145 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:42.145+0000 7faf71361700 1 -- 172.21.15.79:0/1561890553 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}] v 0) v1 -- 0x7faf6c19c790 con 0x7faf6c100da0 2024-05-04T12:15:42.145 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:42.146+0000 7faf53fff700 1 -- 172.21.15.79:0/1561890553 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}]=0 v22)=0 v22) v1 ==== 155+0+0 (secure 0 0 0) 0x7faf60059d60 con 0x7faf6c100da0 2024-05-04T12:15:42.147 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:42.148+0000 7faf71361700 1 -- 172.21.15.79:0/1561890553 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7faf5406c4e0 msgr2=0x7faf5406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:15:42.148 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:42.148+0000 7faf71361700 1 --2- 172.21.15.79:0/1561890553 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7faf5406c4e0 0x7faf5406e9a0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7faf5c00b6f0 tx=0x7faf5c005940 comp rx=0 tx=0).stop 2024-05-04T12:15:42.148 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:42.148+0000 7faf71361700 1 -- 172.21.15.79:0/1561890553 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7faf6c100da0 msgr2=0x7faf6c19b210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:15:42.148 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:42.148+0000 7faf71361700 1 --2- 172.21.15.79:0/1561890553 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7faf6c100da0 0x7faf6c19b210 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7faf6000e920 tx=0x7faf6000ec30 comp rx=0 tx=0).stop 2024-05-04T12:15:42.148 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:42.148+0000 7faf71361700 1 -- 172.21.15.79:0/1561890553 shutdown_connections 2024-05-04T12:15:42.148 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:42.148+0000 7faf71361700 1 --2- 172.21.15.79:0/1561890553 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7faf5406c4e0 0x7faf5406e9a0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:42.148 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:42.148+0000 7faf71361700 1 --2- 172.21.15.79:0/1561890553 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faf6c0ffba0 0x7faf6c069990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:42.148 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:42.148+0000 7faf71361700 1 --2- 172.21.15.79:0/1561890553 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7faf6c100da0 0x7faf6c19b210 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:42.148 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:42.148+0000 7faf71361700 1 -- 172.21.15.79:0/1561890553 >> 172.21.15.79:0/1561890553 conn(0x7faf6c0fb120 msgr2=0x7faf6c103fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:15:42.148 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:42.148+0000 7faf71361700 1 -- 172.21.15.79:0/1561890553 shutdown_connections 2024-05-04T12:15:42.148 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:42.148+0000 7faf71361700 1 -- 172.21.15.79:0/1561890553 wait complete. 2024-05-04T12:15:42.195 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:42 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-04T12:15:42.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:42 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-04T12:15:42.820 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d9b985c4-0a0e-11ef-bc97-c7b262605968 -e sha1=09dbd6bba52d19ba471c33bc15c008e6ad158ea6 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2024-05-04T12:15:43.033 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/d9b985c4-0a0e-11ef-bc97-c7b262605968/mon.smithi079/config 2024-05-04T12:15:43.320 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:43 smithi079 ceph-mon[27222]: pgmap v199: 65 pgs: 65 active+clean; 584 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s wr, 0 op/s 2024-05-04T12:15:43.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:43 smithi146 ceph-mon[35311]: pgmap v199: 65 pgs: 65 active+clean; 584 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s wr, 0 op/s 2024-05-04T12:15:44.674 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:44.672+0000 7f7a904fc700 1 -- 172.21.15.79:0/2171882282 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7a88101f40 msgr2=0x7f7a881023a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:15:44.674 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:44.672+0000 7f7a904fc700 1 --2- 172.21.15.79:0/2171882282 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7a88101f40 0x7f7a881023a0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f7a78009910 tx=0x7f7a78009c20 comp rx=0 tx=0).stop 2024-05-04T12:15:44.674 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:44.672+0000 7f7a904fc700 1 -- 172.21.15.79:0/2171882282 shutdown_connections 2024-05-04T12:15:44.674 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:44.672+0000 7f7a904fc700 1 --2- 172.21.15.79:0/2171882282 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7a88101f40 0x7f7a881023a0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:44.675 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:44.672+0000 7f7a904fc700 1 --2- 172.21.15.79:0/2171882282 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7a880fef60 0x7f7a880ff380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:44.675 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:44.672+0000 7f7a904fc700 1 -- 172.21.15.79:0/2171882282 >> 172.21.15.79:0/2171882282 conn(0x7f7a880fa8c0 msgr2=0x7f7a880fcd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:15:44.675 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:44.672+0000 7f7a904fc700 1 -- 172.21.15.79:0/2171882282 shutdown_connections 2024-05-04T12:15:44.675 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:44.673+0000 7f7a904fc700 1 -- 172.21.15.79:0/2171882282 wait complete. 2024-05-04T12:15:44.675 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:44.673+0000 7f7a904fc700 1 Processor -- start 2024-05-04T12:15:44.675 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:44.673+0000 7f7a904fc700 1 -- start start 2024-05-04T12:15:44.675 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:44.673+0000 7f7a904fc700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7a880fef60 0x7f7a881964e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:15:44.675 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:44.673+0000 7f7a904fc700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7a88101f40 0x7f7a88196a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:15:44.675 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:44.673+0000 7f7a904fc700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a88197040 con 0x7f7a880fef60 2024-05-04T12:15:44.675 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:44.673+0000 7f7a904fc700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a88197180 con 0x7f7a88101f40 2024-05-04T12:15:44.675 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:44.674+0000 7f7a8e298700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7a880fef60 0x7f7a881964e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:15:44.675 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:44.674+0000 7f7a8e298700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7a880fef60 0x7f7a881964e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:51384/0 (socket says 172.21.15.79:51384) 2024-05-04T12:15:44.675 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:44.674+0000 7f7a8e298700 1 -- 172.21.15.79:0/2514825256 learned_addr learned my addr 172.21.15.79:0/2514825256 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:15:44.676 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:44.674+0000 7f7a8da97700 1 --2- 172.21.15.79:0/2514825256 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7a88101f40 0x7f7a88196a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:15:44.676 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:44.674+0000 7f7a8e298700 1 -- 172.21.15.79:0/2514825256 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7a88101f40 msgr2=0x7f7a88196a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:15:44.676 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:44.674+0000 7f7a8e298700 1 --2- 172.21.15.79:0/2514825256 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7a88101f40 0x7f7a88196a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:44.676 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:44.674+0000 7f7a8e298700 1 -- 172.21.15.79:0/2514825256 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a780095f0 con 0x7f7a880fef60 2024-05-04T12:15:44.676 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:44.674+0000 7f7a8e298700 1 --2- 172.21.15.79:0/2514825256 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7a880fef60 0x7f7a881964e0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f7a8400d900 tx=0x7f7a8400dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:15:44.676 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:44.674+0000 7f7a7f7fe700 1 -- 172.21.15.79:0/2514825256 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7a8400b920 con 0x7f7a880fef60 2024-05-04T12:15:44.676 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:44.674+0000 7f7a7f7fe700 1 -- 172.21.15.79:0/2514825256 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f7a84010460 con 0x7f7a880fef60 2024-05-04T12:15:44.676 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:44.674+0000 7f7a7f7fe700 1 -- 172.21.15.79:0/2514825256 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7a8400f590 con 0x7f7a880fef60 2024-05-04T12:15:44.676 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:44.675+0000 7f7a904fc700 1 -- 172.21.15.79:0/2514825256 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a8819bc30 con 0x7f7a880fef60 2024-05-04T12:15:44.676 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:44.675+0000 7f7a904fc700 1 -- 172.21.15.79:0/2514825256 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a8819c180 con 0x7f7a880fef60 2024-05-04T12:15:44.677 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:44.676+0000 7f7a904fc700 1 -- 172.21.15.79:0/2514825256 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a88109800 con 0x7f7a880fef60 2024-05-04T12:15:44.677 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:44.677+0000 7f7a7f7fe700 1 -- 172.21.15.79:0/2514825256 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f7a840105d0 con 0x7f7a880fef60 2024-05-04T12:15:44.677 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:44.677+0000 7f7a7f7fe700 1 --2- 172.21.15.79:0/2514825256 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f7a7406c4e0 0x7f7a7406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:15:44.677 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:44.677+0000 7f7a7f7fe700 1 -- 172.21.15.79:0/2514825256 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f7a8408b6d0 con 0x7f7a880fef60 2024-05-04T12:15:44.677 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:44.677+0000 7f7a8da97700 1 --2- 172.21.15.79:0/2514825256 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f7a7406c4e0 0x7f7a7406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:15:44.677 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:44.678+0000 7f7a8da97700 1 --2- 172.21.15.79:0/2514825256 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f7a7406c4e0 0x7f7a7406e9a0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f7a7800b5c0 tx=0x7f7a78005ee0 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:15:44.680 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:44.681+0000 7f7a7f7fe700 1 -- 172.21.15.79:0/2514825256 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f7a84059ab0 con 0x7f7a880fef60 2024-05-04T12:15:44.886 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:44.887+0000 7f7a904fc700 1 -- 172.21.15.79:0/2514825256 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}] v 0) v1 -- 0x7f7a8819c7f0 con 0x7f7a880fef60 2024-05-04T12:15:44.886 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:44.887+0000 7f7a7f7fe700 1 -- 172.21.15.79:0/2514825256 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}]=0 v22)=0 v22) v1 ==== 163+0+0 (secure 0 0 0) 0x7f7a84059640 con 0x7f7a880fef60 2024-05-04T12:15:44.888 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:44.889+0000 7f7a904fc700 1 -- 172.21.15.79:0/2514825256 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f7a7406c4e0 msgr2=0x7f7a7406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:15:44.888 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:44.889+0000 7f7a904fc700 1 --2- 172.21.15.79:0/2514825256 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f7a7406c4e0 0x7f7a7406e9a0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f7a7800b5c0 tx=0x7f7a78005ee0 comp rx=0 tx=0).stop 2024-05-04T12:15:44.889 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:44.889+0000 7f7a904fc700 1 -- 172.21.15.79:0/2514825256 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7a880fef60 msgr2=0x7f7a881964e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:15:44.889 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:44.889+0000 7f7a904fc700 1 --2- 172.21.15.79:0/2514825256 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7a880fef60 0x7f7a881964e0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f7a8400d900 tx=0x7f7a8400dc10 comp rx=0 tx=0).stop 2024-05-04T12:15:44.889 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:44.889+0000 7f7a904fc700 1 -- 172.21.15.79:0/2514825256 shutdown_connections 2024-05-04T12:15:44.889 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:44.889+0000 7f7a904fc700 1 --2- 172.21.15.79:0/2514825256 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f7a7406c4e0 0x7f7a7406e9a0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:44.889 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:44.889+0000 7f7a904fc700 1 --2- 172.21.15.79:0/2514825256 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7a880fef60 0x7f7a881964e0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:44.889 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:44.889+0000 7f7a904fc700 1 --2- 172.21.15.79:0/2514825256 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7a88101f40 0x7f7a88196a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:44.889 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:44.889+0000 7f7a904fc700 1 -- 172.21.15.79:0/2514825256 >> 172.21.15.79:0/2514825256 conn(0x7f7a880fa8c0 msgr2=0x7f7a88105170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:15:44.889 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:44.889+0000 7f7a904fc700 1 -- 172.21.15.79:0/2514825256 shutdown_connections 2024-05-04T12:15:44.889 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:44.889+0000 7f7a904fc700 1 -- 172.21.15.79:0/2514825256 wait complete. 2024-05-04T12:15:45.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:45 smithi079 ceph-mon[27222]: pgmap v200: 65 pgs: 65 active+clean; 584 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 938 B/s wr, 0 op/s 2024-05-04T12:15:45.526 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:45 smithi146 ceph-mon[35311]: pgmap v200: 65 pgs: 65 active+clean; 584 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 938 B/s wr, 0 op/s 2024-05-04T12:15:45.784 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d9b985c4-0a0e-11ef-bc97-c7b262605968 -e sha1=09dbd6bba52d19ba471c33bc15c008e6ad158ea6 -- bash -c 'ceph config set global log_to_journald false --force' 2024-05-04T12:15:45.995 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/d9b985c4-0a0e-11ef-bc97-c7b262605968/mon.smithi079/config 2024-05-04T12:15:47.437 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:47 smithi079 ceph-mon[27222]: pgmap v201: 65 pgs: 65 active+clean; 584 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 938 B/s wr, 0 op/s 2024-05-04T12:15:47.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:47 smithi146 ceph-mon[35311]: pgmap v201: 65 pgs: 65 active+clean; 584 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 938 B/s wr, 0 op/s 2024-05-04T12:15:47.693 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:47.693+0000 7fa84b629700 1 -- 172.21.15.79:0/1521573466 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa844101ea0 msgr2=0x7fa844102300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:15:47.693 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:47.693+0000 7fa84b629700 1 --2- 172.21.15.79:0/1521573466 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa844101ea0 0x7fa844102300 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fa834009910 tx=0x7fa834009c20 comp rx=0 tx=0).stop 2024-05-04T12:15:47.693 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:47.694+0000 7fa84b629700 1 -- 172.21.15.79:0/1521573466 shutdown_connections 2024-05-04T12:15:47.693 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:47.694+0000 7fa84b629700 1 --2- 172.21.15.79:0/1521573466 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa844101ea0 0x7fa844102300 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:47.693 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:47.694+0000 7fa84b629700 1 --2- 172.21.15.79:0/1521573466 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa844100ca0 0x7fa8441010c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:47.693 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:47.694+0000 7fa84b629700 1 -- 172.21.15.79:0/1521573466 >> 172.21.15.79:0/1521573466 conn(0x7fa8440fc260 msgr2=0x7fa8440fe680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:15:47.695 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:47.694+0000 7fa84b629700 1 -- 172.21.15.79:0/1521573466 shutdown_connections 2024-05-04T12:15:47.695 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:47.694+0000 7fa84b629700 1 -- 172.21.15.79:0/1521573466 wait complete. 2024-05-04T12:15:47.695 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:47.695+0000 7fa84b629700 1 Processor -- start 2024-05-04T12:15:47.695 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:47.695+0000 7fa84b629700 1 -- start start 2024-05-04T12:15:47.695 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:47.695+0000 7fa84b629700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa844100ca0 0x7fa8441964e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:15:47.695 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:47.695+0000 7fa84b629700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa844101ea0 0x7fa844196a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:15:47.695 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:47.695+0000 7fa84b629700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa844197040 con 0x7fa844100ca0 2024-05-04T12:15:47.695 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:47.695+0000 7fa84b629700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa844197180 con 0x7fa844101ea0 2024-05-04T12:15:47.695 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:47.695+0000 7fa8493c5700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa844100ca0 0x7fa8441964e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:15:47.695 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:47.695+0000 7fa8493c5700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa844100ca0 0x7fa8441964e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:51412/0 (socket says 172.21.15.79:51412) 2024-05-04T12:15:47.695 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:47.695+0000 7fa8493c5700 1 -- 172.21.15.79:0/1045204999 learned_addr learned my addr 172.21.15.79:0/1045204999 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:15:47.695 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:47.695+0000 7fa848bc4700 1 --2- 172.21.15.79:0/1045204999 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa844101ea0 0x7fa844196a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:15:47.695 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:47.695+0000 7fa8493c5700 1 -- 172.21.15.79:0/1045204999 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa844101ea0 msgr2=0x7fa844196a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:15:47.696 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:47.695+0000 7fa8493c5700 1 --2- 172.21.15.79:0/1045204999 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa844101ea0 0x7fa844196a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:47.696 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:47.695+0000 7fa8493c5700 1 -- 172.21.15.79:0/1045204999 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8340095f0 con 0x7fa844100ca0 2024-05-04T12:15:47.696 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:47.696+0000 7fa8493c5700 1 --2- 172.21.15.79:0/1045204999 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa844100ca0 0x7fa8441964e0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fa84000d900 tx=0x7fa84000dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:15:47.696 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:47.696+0000 7fa83a7fc700 1 -- 172.21.15.79:0/1045204999 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa8400041f0 con 0x7fa844100ca0 2024-05-04T12:15:47.696 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:47.696+0000 7fa83a7fc700 1 -- 172.21.15.79:0/1045204999 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fa840004350 con 0x7fa844100ca0 2024-05-04T12:15:47.696 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:47.696+0000 7fa84b629700 1 -- 172.21.15.79:0/1045204999 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa84419bc30 con 0x7fa844100ca0 2024-05-04T12:15:47.697 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:47.696+0000 7fa83a7fc700 1 -- 172.21.15.79:0/1045204999 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa840003d20 con 0x7fa844100ca0 2024-05-04T12:15:47.697 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:47.696+0000 7fa84b629700 1 -- 172.21.15.79:0/1045204999 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa84419c180 con 0x7fa844100ca0 2024-05-04T12:15:47.697 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:47.697+0000 7fa84b629700 1 -- 172.21.15.79:0/1045204999 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa844109800 con 0x7fa844100ca0 2024-05-04T12:15:47.697 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:47.698+0000 7fa83a7fc700 1 -- 172.21.15.79:0/1045204999 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7fa840009730 con 0x7fa844100ca0 2024-05-04T12:15:47.697 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:47.698+0000 7fa83a7fc700 1 --2- 172.21.15.79:0/1045204999 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7fa83006c4e0 0x7fa83006e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:15:47.697 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:47.698+0000 7fa83a7fc700 1 -- 172.21.15.79:0/1045204999 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fa84008b750 con 0x7fa844100ca0 2024-05-04T12:15:47.697 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:47.698+0000 7fa848bc4700 1 --2- 172.21.15.79:0/1045204999 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7fa83006c4e0 0x7fa83006e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:15:47.698 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:47.699+0000 7fa848bc4700 1 --2- 172.21.15.79:0/1045204999 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7fa83006c4e0 0x7fa83006e9a0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fa83400b5c0 tx=0x7fa834005ee0 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:15:47.702 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:47.703+0000 7fa83a7fc700 1 -- 172.21.15.79:0/1045204999 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fa840059960 con 0x7fa844100ca0 2024-05-04T12:15:47.900 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:47.900+0000 7fa84b629700 1 -- 172.21.15.79:0/1045204999 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command([{prefix=config set, name=log_to_journald}] v 0) v1 -- 0x7fa8441099f0 con 0x7fa844100ca0 2024-05-04T12:15:47.900 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:47.900+0000 7fa83a7fc700 1 -- 172.21.15.79:0/1045204999 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{prefix=config set, name=log_to_journald}]=0 v22)=0 v22) v1 ==== 135+0+0 (secure 0 0 0) 0x7fa8400594f0 con 0x7fa844100ca0 2024-05-04T12:15:47.902 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:47.902+0000 7fa84b629700 1 -- 172.21.15.79:0/1045204999 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7fa83006c4e0 msgr2=0x7fa83006e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:15:47.902 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:47.902+0000 7fa84b629700 1 --2- 172.21.15.79:0/1045204999 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7fa83006c4e0 0x7fa83006e9a0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fa83400b5c0 tx=0x7fa834005ee0 comp rx=0 tx=0).stop 2024-05-04T12:15:47.902 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:47.902+0000 7fa84b629700 1 -- 172.21.15.79:0/1045204999 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa844100ca0 msgr2=0x7fa8441964e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:15:47.902 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:47.902+0000 7fa84b629700 1 --2- 172.21.15.79:0/1045204999 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa844100ca0 0x7fa8441964e0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fa84000d900 tx=0x7fa84000dc10 comp rx=0 tx=0).stop 2024-05-04T12:15:47.902 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:47.903+0000 7fa84b629700 1 -- 172.21.15.79:0/1045204999 shutdown_connections 2024-05-04T12:15:47.902 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:47.903+0000 7fa84b629700 1 --2- 172.21.15.79:0/1045204999 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7fa83006c4e0 0x7fa83006e9a0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:47.902 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:47.903+0000 7fa84b629700 1 --2- 172.21.15.79:0/1045204999 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa844100ca0 0x7fa8441964e0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:47.902 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:47.903+0000 7fa84b629700 1 --2- 172.21.15.79:0/1045204999 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa844101ea0 0x7fa844196a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:47.902 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:47.903+0000 7fa84b629700 1 -- 172.21.15.79:0/1045204999 >> 172.21.15.79:0/1045204999 conn(0x7fa8440fc260 msgr2=0x7fa8441050d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:15:47.902 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:47.903+0000 7fa84b629700 1 -- 172.21.15.79:0/1045204999 shutdown_connections 2024-05-04T12:15:47.903 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:47.903+0000 7fa84b629700 1 -- 172.21.15.79:0/1045204999 wait complete. 2024-05-04T12:15:48.769 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d9b985c4-0a0e-11ef-bc97-c7b262605968 -e sha1=09dbd6bba52d19ba471c33bc15c008e6ad158ea6 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1' 2024-05-04T12:15:49.047 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/d9b985c4-0a0e-11ef-bc97-c7b262605968/mon.smithi079/config 2024-05-04T12:15:49.322 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:49 smithi079 ceph-mon[27222]: pgmap v202: 65 pgs: 65 active+clean; 584 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 938 B/s wr, 0 op/s 2024-05-04T12:15:49.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:49 smithi146 ceph-mon[35311]: pgmap v202: 65 pgs: 65 active+clean; 584 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 938 B/s wr, 0 op/s 2024-05-04T12:15:50.859 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:50.857+0000 7fe773de5700 1 -- 172.21.15.79:0/828198955 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe76c0ffc10 msgr2=0x7fe76c100030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:15:50.859 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:50.857+0000 7fe773de5700 1 --2- 172.21.15.79:0/828198955 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe76c0ffc10 0x7fe76c100030 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fe768009910 tx=0x7fe768009c20 comp rx=0 tx=0).stop 2024-05-04T12:15:50.859 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:50.857+0000 7fe773de5700 1 -- 172.21.15.79:0/828198955 shutdown_connections 2024-05-04T12:15:50.859 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:50.857+0000 7fe773de5700 1 --2- 172.21.15.79:0/828198955 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe76c100e10 0x7fe76c101270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:50.859 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:50.857+0000 7fe773de5700 1 --2- 172.21.15.79:0/828198955 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe76c0ffc10 0x7fe76c100030 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:50.859 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:50.857+0000 7fe773de5700 1 -- 172.21.15.79:0/828198955 >> 172.21.15.79:0/828198955 conn(0x7fe76c0fb190 msgr2=0x7fe76c0fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:15:50.859 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:50.857+0000 7fe773de5700 1 -- 172.21.15.79:0/828198955 shutdown_connections 2024-05-04T12:15:50.859 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:50.857+0000 7fe773de5700 1 -- 172.21.15.79:0/828198955 wait complete. 2024-05-04T12:15:50.859 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:50.858+0000 7fe773de5700 1 Processor -- start 2024-05-04T12:15:50.859 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:50.858+0000 7fe773de5700 1 -- start start 2024-05-04T12:15:50.860 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:50.858+0000 7fe773de5700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe76c0ffc10 0x7fe76c1964d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:15:50.860 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:50.858+0000 7fe773de5700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe76c100e10 0x7fe76c196a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:15:50.860 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:50.858+0000 7fe773de5700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe76c197030 con 0x7fe76c0ffc10 2024-05-04T12:15:50.860 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:50.858+0000 7fe773de5700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe76c197170 con 0x7fe76c100e10 2024-05-04T12:15:50.860 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:50.858+0000 7fe771b81700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe76c0ffc10 0x7fe76c1964d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:15:50.860 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:50.858+0000 7fe771b81700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe76c0ffc10 0x7fe76c1964d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:51432/0 (socket says 172.21.15.79:51432) 2024-05-04T12:15:50.860 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:50.858+0000 7fe771b81700 1 -- 172.21.15.79:0/3175611864 learned_addr learned my addr 172.21.15.79:0/3175611864 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:15:50.860 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:50.858+0000 7fe771380700 1 --2- 172.21.15.79:0/3175611864 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe76c100e10 0x7fe76c196a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:15:50.860 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:50.859+0000 7fe771b81700 1 -- 172.21.15.79:0/3175611864 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe76c100e10 msgr2=0x7fe76c196a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:15:50.860 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:50.859+0000 7fe771b81700 1 --2- 172.21.15.79:0/3175611864 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe76c100e10 0x7fe76c196a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:50.860 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:50.859+0000 7fe771b81700 1 -- 172.21.15.79:0/3175611864 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7680095f0 con 0x7fe76c0ffc10 2024-05-04T12:15:50.860 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:50.859+0000 7fe771b81700 1 --2- 172.21.15.79:0/3175611864 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe76c0ffc10 0x7fe76c1964d0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fe76800bda0 tx=0x7fe76800bdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:15:50.860 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:50.859+0000 7fe762ffd700 1 -- 172.21.15.79:0/3175611864 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe76801d070 con 0x7fe76c0ffc10 2024-05-04T12:15:50.860 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:50.859+0000 7fe773de5700 1 -- 172.21.15.79:0/3175611864 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe76c19bbc0 con 0x7fe76c0ffc10 2024-05-04T12:15:50.861 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:50.859+0000 7fe762ffd700 1 -- 172.21.15.79:0/3175611864 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fe76800f460 con 0x7fe76c0ffc10 2024-05-04T12:15:50.861 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:50.859+0000 7fe762ffd700 1 -- 172.21.15.79:0/3175611864 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe768021640 con 0x7fe76c0ffc10 2024-05-04T12:15:50.861 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:50.859+0000 7fe773de5700 1 -- 172.21.15.79:0/3175611864 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe76c19c0b0 con 0x7fe76c0ffc10 2024-05-04T12:15:50.861 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:50.860+0000 7fe773de5700 1 -- 172.21.15.79:0/3175611864 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe76c108700 con 0x7fe76c0ffc10 2024-05-04T12:15:50.862 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:50.861+0000 7fe762ffd700 1 -- 172.21.15.79:0/3175611864 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7fe76800f5d0 con 0x7fe76c0ffc10 2024-05-04T12:15:50.862 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:50.861+0000 7fe762ffd700 1 --2- 172.21.15.79:0/3175611864 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7fe75806c4e0 0x7fe75806e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:15:50.862 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:50.861+0000 7fe762ffd700 1 -- 172.21.15.79:0/3175611864 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fe76808ca80 con 0x7fe76c0ffc10 2024-05-04T12:15:50.862 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:50.861+0000 7fe771380700 1 --2- 172.21.15.79:0/3175611864 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7fe75806c4e0 0x7fe75806e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:15:50.862 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:50.862+0000 7fe771380700 1 --2- 172.21.15.79:0/3175611864 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7fe75806c4e0 0x7fe75806e9a0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fe75c007be0 tx=0x7fe75c0073d0 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:15:50.865 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:50.866+0000 7fe762ffd700 1 -- 172.21.15.79:0/3175611864 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fe76805ad10 con 0x7fe76c0ffc10 2024-05-04T12:15:51.122 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:51.123+0000 7fe773de5700 1 -- 172.21.15.79:0/3175611864 --> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] -- mgr_command(tid 0: {"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:09dbd6bba52d19ba471c33bc15c008e6ad158ea6", "target": ["mon-mgr", ""]}) v1 -- 0x7fe76c068d30 con 0x7fe75806c4e0 2024-05-04T12:15:51.128 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:51.129+0000 7fe762ffd700 1 -- 172.21.15.79:0/3175611864 <== mgr.14219 v2:172.21.15.79:6800/402764217 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+89 (secure 0 0 0) 0x7fe76c068d30 con 0x7fe75806c4e0 2024-05-04T12:15:51.128 INFO:teuthology.orchestra.run.smithi079.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:09dbd6bba52d19ba471c33bc15c008e6ad158ea6 2024-05-04T12:15:51.131 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:51.131+0000 7fe773de5700 1 -- 172.21.15.79:0/3175611864 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7fe75806c4e0 msgr2=0x7fe75806e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:15:51.131 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:51.131+0000 7fe773de5700 1 --2- 172.21.15.79:0/3175611864 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7fe75806c4e0 0x7fe75806e9a0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fe75c007be0 tx=0x7fe75c0073d0 comp rx=0 tx=0).stop 2024-05-04T12:15:51.131 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:51.131+0000 7fe773de5700 1 -- 172.21.15.79:0/3175611864 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe76c0ffc10 msgr2=0x7fe76c1964d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:15:51.131 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:51.131+0000 7fe773de5700 1 --2- 172.21.15.79:0/3175611864 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe76c0ffc10 0x7fe76c1964d0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fe76800bda0 tx=0x7fe76800bdd0 comp rx=0 tx=0).stop 2024-05-04T12:15:51.131 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:51.131+0000 7fe773de5700 1 -- 172.21.15.79:0/3175611864 shutdown_connections 2024-05-04T12:15:51.131 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:51.131+0000 7fe773de5700 1 --2- 172.21.15.79:0/3175611864 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7fe75806c4e0 0x7fe75806e9a0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:51.131 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:51.131+0000 7fe773de5700 1 --2- 172.21.15.79:0/3175611864 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe76c0ffc10 0x7fe76c1964d0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:51.131 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:51.131+0000 7fe773de5700 1 --2- 172.21.15.79:0/3175611864 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe76c100e10 0x7fe76c196a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:51.131 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:51.131+0000 7fe773de5700 1 -- 172.21.15.79:0/3175611864 >> 172.21.15.79:0/3175611864 conn(0x7fe76c0fb190 msgr2=0x7fe76c068580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:15:51.131 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:51.131+0000 7fe773de5700 1 -- 172.21.15.79:0/3175611864 shutdown_connections 2024-05-04T12:15:51.131 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:51.131+0000 7fe773de5700 1 -- 172.21.15.79:0/3175611864 wait complete. 2024-05-04T12:15:51.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:51 smithi079 ceph-mon[27222]: pgmap v203: 65 pgs: 65 active+clean; 584 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 682 B/s wr, 0 op/s 2024-05-04T12:15:51.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:51 smithi146 ceph-mon[35311]: pgmap v203: 65 pgs: 65 active+clean; 584 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 682 B/s wr, 0 op/s 2024-05-04T12:15:51.945 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-05-04T12:15:51.946 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi079.front.sepia.ceph.com 2024-05-04T12:15:51.946 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d9b985c4-0a0e-11ef-bc97-c7b262605968 -e sha1=09dbd6bba52d19ba471c33bc15c008e6ad158ea6 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done' 2024-05-04T12:15:52.183 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/d9b985c4-0a0e-11ef-bc97-c7b262605968/mon.smithi079/config 2024-05-04T12:15:52.195 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:52 smithi079 ceph-mon[27222]: from='client.14554 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:09dbd6bba52d19ba471c33bc15c008e6ad158ea6", "target": ["mon-mgr", ""]}]: dispatch 2024-05-04T12:15:52.195 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:52 smithi079 ceph-mon[27222]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:09dbd6bba52d19ba471c33bc15c008e6ad158ea6 2024-05-04T12:15:52.195 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:52 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:15:52.195 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:52 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-04T12:15:52.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:52 smithi146 ceph-mon[35311]: from='client.14554 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:09dbd6bba52d19ba471c33bc15c008e6ad158ea6", "target": ["mon-mgr", ""]}]: dispatch 2024-05-04T12:15:52.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:52 smithi146 ceph-mon[35311]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:09dbd6bba52d19ba471c33bc15c008e6ad158ea6 2024-05-04T12:15:52.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:52 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:15:52.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:52 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-04T12:15:53.302 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:53 smithi079 ceph-mon[27222]: pgmap v204: 65 pgs: 65 active+clean; 584 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 682 B/s wr, 0 op/s 2024-05-04T12:15:53.302 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:53 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-04T12:15:53.302 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:53 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-04T12:15:53.303 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:53 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:15:53.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:53 smithi146 ceph-mon[35311]: pgmap v204: 65 pgs: 65 active+clean; 584 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 682 B/s wr, 0 op/s 2024-05-04T12:15:53.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:53 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-04T12:15:53.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:53 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-04T12:15:53.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:53 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:15:53.989 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:53.989+0000 7f4965664700 1 -- 172.21.15.79:0/3235010078 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f49600ffc10 msgr2=0x7f4960100030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:15:53.989 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:53.989+0000 7f4965664700 1 --2- 172.21.15.79:0/3235010078 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f49600ffc10 0x7f4960100030 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f4950009910 tx=0x7f4950009c20 comp rx=0 tx=0).stop 2024-05-04T12:15:53.989 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:53.989+0000 7f4965664700 1 -- 172.21.15.79:0/3235010078 shutdown_connections 2024-05-04T12:15:53.989 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:53.989+0000 7f4965664700 1 --2- 172.21.15.79:0/3235010078 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4960100e10 0x7f4960101270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:53.989 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:53.989+0000 7f4965664700 1 --2- 172.21.15.79:0/3235010078 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f49600ffc10 0x7f4960100030 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:53.989 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:53.989+0000 7f4965664700 1 -- 172.21.15.79:0/3235010078 >> 172.21.15.79:0/3235010078 conn(0x7f49600fb190 msgr2=0x7f49600fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:15:53.989 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:53.989+0000 7f4965664700 1 -- 172.21.15.79:0/3235010078 shutdown_connections 2024-05-04T12:15:53.989 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:53.989+0000 7f4965664700 1 -- 172.21.15.79:0/3235010078 wait complete. 2024-05-04T12:15:53.990 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:53.990+0000 7f4965664700 1 Processor -- start 2024-05-04T12:15:53.990 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:53.990+0000 7f4965664700 1 -- start start 2024-05-04T12:15:53.991 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:53.990+0000 7f4965664700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f49600ffc10 0x7f49600690e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:15:53.991 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:53.990+0000 7f4965664700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4960100e10 0x7f496019d630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:15:53.991 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:53.990+0000 7f4965664700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49601970a0 con 0x7f49600ffc10 2024-05-04T12:15:53.991 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:53.990+0000 7f4965664700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f496019db70 con 0x7f4960100e10 2024-05-04T12:15:53.991 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:53.990+0000 7f495effd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f49600ffc10 0x7f49600690e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:15:53.991 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:53.990+0000 7f495effd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f49600ffc10 0x7f49600690e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:58848/0 (socket says 172.21.15.79:58848) 2024-05-04T12:15:53.991 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:53.990+0000 7f495effd700 1 -- 172.21.15.79:0/1132186470 learned_addr learned my addr 172.21.15.79:0/1132186470 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:15:53.991 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:53.990+0000 7f495e7fc700 1 --2- 172.21.15.79:0/1132186470 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4960100e10 0x7f496019d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:15:53.992 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:53.991+0000 7f495effd700 1 -- 172.21.15.79:0/1132186470 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4960100e10 msgr2=0x7f496019d630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:15:53.992 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:53.991+0000 7f495effd700 1 --2- 172.21.15.79:0/1132186470 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4960100e10 0x7f496019d630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:53.992 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:53.991+0000 7f495effd700 1 -- 172.21.15.79:0/1132186470 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49500095f0 con 0x7f49600ffc10 2024-05-04T12:15:53.992 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:53.991+0000 7f495effd700 1 --2- 172.21.15.79:0/1132186470 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f49600ffc10 0x7f49600690e0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f495000f690 tx=0x7f495000f6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:15:53.992 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:53.991+0000 7f4947fff700 1 -- 172.21.15.79:0/1132186470 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f495001d070 con 0x7f49600ffc10 2024-05-04T12:15:53.992 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:53.991+0000 7f4947fff700 1 -- 172.21.15.79:0/1132186470 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f4950005dc0 con 0x7f49600ffc10 2024-05-04T12:15:53.992 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:53.991+0000 7f4965664700 1 -- 172.21.15.79:0/1132186470 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f496019ddc0 con 0x7f49600ffc10 2024-05-04T12:15:53.992 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:53.991+0000 7f4947fff700 1 -- 172.21.15.79:0/1132186470 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4950022470 con 0x7f49600ffc10 2024-05-04T12:15:53.992 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:53.991+0000 7f4965664700 1 -- 172.21.15.79:0/1132186470 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f496019e2b0 con 0x7f49600ffc10 2024-05-04T12:15:53.993 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:53.992+0000 7f4947fff700 1 -- 172.21.15.79:0/1132186470 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f49500038a0 con 0x7f49600ffc10 2024-05-04T12:15:53.993 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:53.992+0000 7f4965664700 1 -- 172.21.15.79:0/1132186470 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4960108700 con 0x7f49600ffc10 2024-05-04T12:15:53.993 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:53.993+0000 7f4947fff700 1 --2- 172.21.15.79:0/1132186470 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f494806c4e0 0x7f494806e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:15:53.993 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:53.993+0000 7f4947fff700 1 -- 172.21.15.79:0/1132186470 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f495008ccb0 con 0x7f49600ffc10 2024-05-04T12:15:53.993 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:53.993+0000 7f495e7fc700 1 --2- 172.21.15.79:0/1132186470 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f494806c4e0 0x7f494806e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:15:53.993 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:53.993+0000 7f495e7fc700 1 --2- 172.21.15.79:0/1132186470 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f494806c4e0 0x7f494806e9a0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f4960069660 tx=0x7f4954009260 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:15:53.997 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:53.998+0000 7f4947fff700 1 -- 172.21.15.79:0/1132186470 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f495005af40 con 0x7f49600ffc10 2024-05-04T12:15:54.233 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:54.234+0000 7f4965664700 1 -- 172.21.15.79:0/1132186470 --> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f4960105760 con 0x7f494806c4e0 2024-05-04T12:15:54.234 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:54.235+0000 7f4947fff700 1 -- 172.21.15.79:0/1132186470 <== mgr.14219 v2:172.21.15.79:6800/402764217 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+329 (secure 0 0 0) 0x7f4960105760 con 0x7f494806c4e0 2024-05-04T12:15:54.237 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:54.237+0000 7f4965664700 1 -- 172.21.15.79:0/1132186470 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f494806c4e0 msgr2=0x7f494806e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:15:54.237 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:54.237+0000 7f4965664700 1 --2- 172.21.15.79:0/1132186470 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f494806c4e0 0x7f494806e9a0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f4960069660 tx=0x7f4954009260 comp rx=0 tx=0).stop 2024-05-04T12:15:54.237 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:54.237+0000 7f4965664700 1 -- 172.21.15.79:0/1132186470 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f49600ffc10 msgr2=0x7f49600690e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:15:54.237 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:54.237+0000 7f4965664700 1 --2- 172.21.15.79:0/1132186470 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f49600ffc10 0x7f49600690e0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f495000f690 tx=0x7f495000f6c0 comp rx=0 tx=0).stop 2024-05-04T12:15:54.237 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:54.237+0000 7f4965664700 1 -- 172.21.15.79:0/1132186470 shutdown_connections 2024-05-04T12:15:54.237 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:54.237+0000 7f4965664700 1 --2- 172.21.15.79:0/1132186470 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f494806c4e0 0x7f494806e9a0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:54.237 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:54.237+0000 7f4965664700 1 --2- 172.21.15.79:0/1132186470 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f49600ffc10 0x7f49600690e0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:54.237 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:54.237+0000 7f4965664700 1 --2- 172.21.15.79:0/1132186470 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4960100e10 0x7f496019d630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:54.238 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:54.237+0000 7f4965664700 1 -- 172.21.15.79:0/1132186470 >> 172.21.15.79:0/1132186470 conn(0x7f49600fb190 msgr2=0x7f4960104040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:15:54.238 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:54.237+0000 7f4965664700 1 -- 172.21.15.79:0/1132186470 shutdown_connections 2024-05-04T12:15:54.238 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:54.237+0000 7f4965664700 1 -- 172.21.15.79:0/1132186470 wait complete. 2024-05-04T12:15:54.247 INFO:teuthology.orchestra.run.smithi079.stdout:true 2024-05-04T12:15:54.356 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:54.357+0000 7fcca14d8700 1 -- 172.21.15.79:0/2109527018 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcc9c104070 msgr2=0x7fcc9c106460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:15:54.357 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:54.357+0000 7fcca14d8700 1 --2- 172.21.15.79:0/2109527018 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcc9c104070 0x7fcc9c106460 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fcc90009910 tx=0x7fcc90009c20 comp rx=0 tx=0).stop 2024-05-04T12:15:54.357 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:54.357+0000 7fcca14d8700 1 -- 172.21.15.79:0/2109527018 shutdown_connections 2024-05-04T12:15:54.357 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:54.357+0000 7fcca14d8700 1 --2- 172.21.15.79:0/2109527018 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcc9c104070 0x7fcc9c106460 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:54.357 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:54.357+0000 7fcca14d8700 1 --2- 172.21.15.79:0/2109527018 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcc9c101740 0x7fcc9c103b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:54.357 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:54.357+0000 7fcca14d8700 1 -- 172.21.15.79:0/2109527018 >> 172.21.15.79:0/2109527018 conn(0x7fcc9c0fb150 msgr2=0x7fcc9c0fd590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:15:54.357 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:54.357+0000 7fcca14d8700 1 -- 172.21.15.79:0/2109527018 shutdown_connections 2024-05-04T12:15:54.357 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:54.357+0000 7fcca14d8700 1 -- 172.21.15.79:0/2109527018 wait complete. 2024-05-04T12:15:54.357 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:54.357+0000 7fcca14d8700 1 Processor -- start 2024-05-04T12:15:54.359 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:54.358+0000 7fcca14d8700 1 -- start start 2024-05-04T12:15:54.359 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:54.358+0000 7fcca14d8700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcc9c101740 0x7fcc9c1986a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:15:54.359 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:54.358+0000 7fcca14d8700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcc9c104070 0x7fcc9c198be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:15:54.359 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:54.358+0000 7fcca14d8700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc9c199200 con 0x7fcc9c101740 2024-05-04T12:15:54.359 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:54.358+0000 7fcca14d8700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc9c199340 con 0x7fcc9c104070 2024-05-04T12:15:54.359 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:54.358+0000 7fcc9affd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcc9c101740 0x7fcc9c1986a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:15:54.359 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:54.358+0000 7fcc9affd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcc9c101740 0x7fcc9c1986a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:58868/0 (socket says 172.21.15.79:58868) 2024-05-04T12:15:54.359 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:54.358+0000 7fcc9affd700 1 -- 172.21.15.79:0/3351485178 learned_addr learned my addr 172.21.15.79:0/3351485178 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:15:54.359 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:54.358+0000 7fcc9a7fc700 1 --2- 172.21.15.79:0/3351485178 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcc9c104070 0x7fcc9c198be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:15:54.360 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:54.358+0000 7fcc9affd700 1 -- 172.21.15.79:0/3351485178 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcc9c104070 msgr2=0x7fcc9c198be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:15:54.360 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:54.358+0000 7fcc9affd700 1 --2- 172.21.15.79:0/3351485178 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcc9c104070 0x7fcc9c198be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:54.360 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:54.358+0000 7fcc9affd700 1 -- 172.21.15.79:0/3351485178 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc900095f0 con 0x7fcc9c101740 2024-05-04T12:15:54.360 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:54.358+0000 7fcc9a7fc700 1 --2- 172.21.15.79:0/3351485178 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcc9c104070 0x7fcc9c198be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-04T12:15:54.360 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:54.358+0000 7fcc9affd700 1 --2- 172.21.15.79:0/3351485178 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcc9c101740 0x7fcc9c1986a0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fcc8c00e940 tx=0x7fcc8c00ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:15:54.360 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:54.359+0000 7fcc83fff700 1 -- 172.21.15.79:0/3351485178 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcc8c00cba0 con 0x7fcc9c101740 2024-05-04T12:15:54.360 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:54.359+0000 7fcc83fff700 1 -- 172.21.15.79:0/3351485178 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fcc8c00ee90 con 0x7fcc9c101740 2024-05-04T12:15:54.360 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:54.359+0000 7fcca14d8700 1 -- 172.21.15.79:0/3351485178 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc9c19ddf0 con 0x7fcc9c101740 2024-05-04T12:15:54.360 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:54.359+0000 7fcca14d8700 1 -- 172.21.15.79:0/3351485178 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc9c19e270 con 0x7fcc9c101740 2024-05-04T12:15:54.360 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:54.359+0000 7fcc83fff700 1 -- 172.21.15.79:0/3351485178 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcc8c005650 con 0x7fcc9c101740 2024-05-04T12:15:54.360 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:54.360+0000 7fcca14d8700 1 -- 172.21.15.79:0/3351485178 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc9c0fcd20 con 0x7fcc9c101740 2024-05-04T12:15:54.361 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:54.360+0000 7fcc83fff700 1 -- 172.21.15.79:0/3351485178 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7fcc8c004750 con 0x7fcc9c101740 2024-05-04T12:15:54.361 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:54.361+0000 7fcc83fff700 1 --2- 172.21.15.79:0/3351485178 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7fcc8406c4e0 0x7fcc8406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:15:54.361 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:54.361+0000 7fcc83fff700 1 -- 172.21.15.79:0/3351485178 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fcc8c014070 con 0x7fcc9c101740 2024-05-04T12:15:54.361 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:54.361+0000 7fcc9a7fc700 1 --2- 172.21.15.79:0/3351485178 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7fcc8406c4e0 0x7fcc8406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:15:54.361 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:54.361+0000 7fcc9a7fc700 1 --2- 172.21.15.79:0/3351485178 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7fcc8406c4e0 0x7fcc8406e9a0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fcc90009fd0 tx=0x7fcc90005940 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:15:54.364 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:54.365+0000 7fcc83fff700 1 -- 172.21.15.79:0/3351485178 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fcc8c059d60 con 0x7fcc9c101740 2024-05-04T12:15:54.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:54 smithi079 ceph-mon[27222]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:09dbd6bba52d19ba471c33bc15c008e6ad158ea6 2024-05-04T12:15:54.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:54 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:15:54.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:54 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-04T12:15:54.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:54 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-04T12:15:54.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:54 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-04T12:15:54.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:54 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:15:54.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:54 smithi146 ceph-mon[35311]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:09dbd6bba52d19ba471c33bc15c008e6ad158ea6 2024-05-04T12:15:54.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:54 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:15:54.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:54 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-04T12:15:54.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:54 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-04T12:15:54.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:54 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-04T12:15:54.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:54 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:15:54.614 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:54.615+0000 7fcca14d8700 1 -- 172.21.15.79:0/3351485178 --> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fcc9c0611d0 con 0x7fcc8406c4e0 2024-05-04T12:15:54.615 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:54.616+0000 7fcc83fff700 1 -- 172.21.15.79:0/3351485178 <== mgr.14219 v2:172.21.15.79:6800/402764217 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+329 (secure 0 0 0) 0x7fcc9c0611d0 con 0x7fcc8406c4e0 2024-05-04T12:15:54.618 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:54.618+0000 7fcca14d8700 1 -- 172.21.15.79:0/3351485178 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7fcc8406c4e0 msgr2=0x7fcc8406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:15:54.618 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:54.618+0000 7fcca14d8700 1 --2- 172.21.15.79:0/3351485178 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7fcc8406c4e0 0x7fcc8406e9a0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fcc90009fd0 tx=0x7fcc90005940 comp rx=0 tx=0).stop 2024-05-04T12:15:54.618 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:54.618+0000 7fcca14d8700 1 -- 172.21.15.79:0/3351485178 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcc9c101740 msgr2=0x7fcc9c1986a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:15:54.618 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:54.618+0000 7fcca14d8700 1 --2- 172.21.15.79:0/3351485178 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcc9c101740 0x7fcc9c1986a0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fcc8c00e940 tx=0x7fcc8c00ec50 comp rx=0 tx=0).stop 2024-05-04T12:15:54.618 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:54.618+0000 7fcca14d8700 1 -- 172.21.15.79:0/3351485178 shutdown_connections 2024-05-04T12:15:54.618 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:54.618+0000 7fcca14d8700 1 --2- 172.21.15.79:0/3351485178 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7fcc8406c4e0 0x7fcc8406e9a0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:54.618 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:54.618+0000 7fcca14d8700 1 --2- 172.21.15.79:0/3351485178 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcc9c101740 0x7fcc9c1986a0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:54.618 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:54.618+0000 7fcca14d8700 1 --2- 172.21.15.79:0/3351485178 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcc9c104070 0x7fcc9c198be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:54.618 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:54.618+0000 7fcca14d8700 1 -- 172.21.15.79:0/3351485178 >> 172.21.15.79:0/3351485178 conn(0x7fcc9c0fb150 msgr2=0x7fcc9c100fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:15:54.618 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:54.618+0000 7fcca14d8700 1 -- 172.21.15.79:0/3351485178 shutdown_connections 2024-05-04T12:15:54.619 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:54.618+0000 7fcca14d8700 1 -- 172.21.15.79:0/3351485178 wait complete. 2024-05-04T12:15:54.629 INFO:teuthology.orchestra.run.smithi079.stdout:"Error: UPGRADE_FAILED_PULL: Upgrade: failed to pull target image" 2024-05-04T12:15:55.365 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:55 smithi079 ceph-mon[27222]: pgmap v205: 65 pgs: 65 active+clean; 584 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2024-05-04T12:15:55.365 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:55 smithi079 ceph-mon[27222]: Upgrade: Paused due to UPGRADE_FAILED_PULL: Upgrade: failed to pull target image 2024-05-04T12:15:55.366 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:55 smithi079 ceph-mon[27222]: pgmap v206: 65 pgs: 65 active+clean; 584 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-04T12:15:55.366 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:55 smithi079 ceph-mon[27222]: Health check failed: Upgrade: failed to pull target image (UPGRADE_FAILED_PULL) 2024-05-04T12:15:55.387 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d9b985c4-0a0e-11ef-bc97-c7b262605968 -e sha1=09dbd6bba52d19ba471c33bc15c008e6ad158ea6 -- bash -c 'ceph orch ps' 2024-05-04T12:15:55.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:55 smithi146 ceph-mon[35311]: pgmap v205: 65 pgs: 65 active+clean; 584 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2024-05-04T12:15:55.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:55 smithi146 ceph-mon[35311]: Upgrade: Paused due to UPGRADE_FAILED_PULL: Upgrade: failed to pull target image 2024-05-04T12:15:55.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:55 smithi146 ceph-mon[35311]: pgmap v206: 65 pgs: 65 active+clean; 584 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-04T12:15:55.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:55 smithi146 ceph-mon[35311]: Health check failed: Upgrade: failed to pull target image (UPGRADE_FAILED_PULL) 2024-05-04T12:15:55.641 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/d9b985c4-0a0e-11ef-bc97-c7b262605968/mon.smithi079/config 2024-05-04T12:15:56.406 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:56 smithi079 ceph-mon[27222]: from='client.14558 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-04T12:15:56.406 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:56 smithi079 ceph-mon[27222]: from='client.14562 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-04T12:15:56.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:56 smithi146 ceph-mon[35311]: from='client.14558 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-04T12:15:56.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:56 smithi146 ceph-mon[35311]: from='client.14562 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-04T12:15:57.283 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:57.283+0000 7f161e27b700 1 -- 172.21.15.79:0/1620366372 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f16181022a0 msgr2=0x7f16181026c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:15:57.284 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:57.283+0000 7f161e27b700 1 --2- 172.21.15.79:0/1620366372 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f16181022a0 0x7f16181026c0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f1608009910 tx=0x7f1608009c20 comp rx=0 tx=0).stop 2024-05-04T12:15:57.284 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:57.284+0000 7f161e27b700 1 -- 172.21.15.79:0/1620366372 shutdown_connections 2024-05-04T12:15:57.284 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:57.284+0000 7f161e27b700 1 --2- 172.21.15.79:0/1620366372 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f16180fed20 0x7f16180ff1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:57.284 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:57.284+0000 7f161e27b700 1 --2- 172.21.15.79:0/1620366372 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f16181022a0 0x7f16181026c0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:57.284 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:57.284+0000 7f161e27b700 1 -- 172.21.15.79:0/1620366372 >> 172.21.15.79:0/1620366372 conn(0x7f16180fa8c0 msgr2=0x7f16180fcd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:15:57.284 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:57.284+0000 7f161e27b700 1 -- 172.21.15.79:0/1620366372 shutdown_connections 2024-05-04T12:15:57.284 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:57.284+0000 7f161e27b700 1 -- 172.21.15.79:0/1620366372 wait complete. 2024-05-04T12:15:57.286 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:57.285+0000 7f161e27b700 1 Processor -- start 2024-05-04T12:15:57.286 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:57.285+0000 7f161e27b700 1 -- start start 2024-05-04T12:15:57.286 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:57.285+0000 7f161e27b700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f16180fed20 0x7f1618192800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:15:57.286 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:57.285+0000 7f161e27b700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f16181022a0 0x7f1618192d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:15:57.286 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:57.285+0000 7f161e27b700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f161818f2f0 con 0x7f16180fed20 2024-05-04T12:15:57.286 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:57.285+0000 7f161e27b700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f161818f460 con 0x7f16181022a0 2024-05-04T12:15:57.286 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:57.285+0000 7f1617fff700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f16180fed20 0x7f1618192800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:15:57.286 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:57.285+0000 7f1617fff700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f16180fed20 0x7f1618192800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:58888/0 (socket says 172.21.15.79:58888) 2024-05-04T12:15:57.286 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:57.285+0000 7f1617fff700 1 -- 172.21.15.79:0/2791341803 learned_addr learned my addr 172.21.15.79:0/2791341803 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:15:57.286 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:57.286+0000 7f1617fff700 1 -- 172.21.15.79:0/2791341803 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f16181022a0 msgr2=0x7f1618192d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:15:57.287 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:57.286+0000 7f1617fff700 1 --2- 172.21.15.79:0/2791341803 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f16181022a0 0x7f1618192d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:57.287 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:57.286+0000 7f1617fff700 1 -- 172.21.15.79:0/2791341803 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16080095f0 con 0x7f16180fed20 2024-05-04T12:15:57.287 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:57.286+0000 7f1617fff700 1 --2- 172.21.15.79:0/2791341803 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f16180fed20 0x7f1618192800 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f160800bda0 tx=0x7f160800be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:15:57.287 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:57.286+0000 7f16157fa700 1 -- 172.21.15.79:0/2791341803 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f160801d070 con 0x7f16180fed20 2024-05-04T12:15:57.287 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:57.286+0000 7f161e27b700 1 -- 172.21.15.79:0/2791341803 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f161818f6e0 con 0x7f16180fed20 2024-05-04T12:15:57.287 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:57.286+0000 7f161e27b700 1 -- 172.21.15.79:0/2791341803 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f161818fbd0 con 0x7f16180fed20 2024-05-04T12:15:57.287 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:57.286+0000 7f16157fa700 1 -- 172.21.15.79:0/2791341803 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f160800f460 con 0x7f16180fed20 2024-05-04T12:15:57.287 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:57.286+0000 7f16157fa700 1 -- 172.21.15.79:0/2791341803 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1608021640 con 0x7f16180fed20 2024-05-04T12:15:57.288 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:57.288+0000 7f161e27b700 1 -- 172.21.15.79:0/2791341803 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1604005320 con 0x7f16180fed20 2024-05-04T12:15:57.288 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:57.288+0000 7f16157fa700 1 -- 172.21.15.79:0/2791341803 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f1608003a60 con 0x7f16180fed20 2024-05-04T12:15:57.288 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:57.289+0000 7f16157fa700 1 --2- 172.21.15.79:0/2791341803 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f160006c4e0 0x7f160006e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:15:57.288 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:57.289+0000 7f16157fa700 1 -- 172.21.15.79:0/2791341803 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f160808cab0 con 0x7f16180fed20 2024-05-04T12:15:57.289 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:57.289+0000 7f16177fe700 1 --2- 172.21.15.79:0/2791341803 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f160006c4e0 0x7f160006e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:15:57.289 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:57.290+0000 7f16177fe700 1 --2- 172.21.15.79:0/2791341803 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f160006c4e0 0x7f160006e9a0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f160c005fd0 tx=0x7f160c005f60 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:15:57.293 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:57.294+0000 7f16157fa700 1 -- 172.21.15.79:0/2791341803 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f160808ee60 con 0x7f16180fed20 2024-05-04T12:15:57.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:57 smithi079 ceph-mon[27222]: pgmap v207: 65 pgs: 65 active+clean; 584 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-04T12:15:57.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:57 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:15:57.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:57 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-04T12:15:57.517 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:57.518+0000 7f161e27b700 1 -- 172.21.15.79:0/2791341803 --> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f1604000bf0 con 0x7f160006c4e0 2024-05-04T12:15:57.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:57 smithi146 ceph-mon[35311]: pgmap v207: 65 pgs: 65 active+clean; 584 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-04T12:15:57.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:57 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' 2024-05-04T12:15:57.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:57 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-04T12:15:57.535 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:57.536+0000 7f16157fa700 1 -- 172.21.15.79:0/2791341803 <== mgr.14219 v2:172.21.15.79:6800/402764217 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3480 (secure 0 0 0) 0x7f1604000bf0 con 0x7f160006c4e0 2024-05-04T12:15:57.536 INFO:teuthology.orchestra.run.smithi079.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-05-04T12:15:57.536 INFO:teuthology.orchestra.run.smithi079.stdout:alertmanager.smithi079 smithi079 *:9093,9094 running (4m) 28s ago 6m 27.0M - 0.25.0 c8568f914cd2 65445b7ae71b 2024-05-04T12:15:57.536 INFO:teuthology.orchestra.run.smithi079.stdout:ceph-exporter.smithi079 smithi079 running (6m) 28s ago 6m 7448k - 18.2.1 5be31c24972a f691c99a7c23 2024-05-04T12:15:57.536 INFO:teuthology.orchestra.run.smithi079.stdout:ceph-exporter.smithi146 smithi146 running (5m) 33s ago 5m 7176k - 18.2.1 5be31c24972a 71288aa0ec98 2024-05-04T12:15:57.536 INFO:teuthology.orchestra.run.smithi079.stdout:crash.smithi079 smithi079 running (6m) 28s ago 6m 6656k - 18.2.1 5be31c24972a 6ce9cc3a3e35 2024-05-04T12:15:57.536 INFO:teuthology.orchestra.run.smithi079.stdout:crash.smithi146 smithi146 running (5m) 33s ago 5m 6660k - 18.2.1 5be31c24972a c42b91d1d7da 2024-05-04T12:15:57.536 INFO:teuthology.orchestra.run.smithi079.stdout:grafana.smithi079 smithi079 *:3000 running (4m) 28s ago 6m 87.9M - 9.4.7 954c08fa6188 a6fce20482ec 2024-05-04T12:15:57.536 INFO:teuthology.orchestra.run.smithi079.stdout:mds.cephfs.smithi079.hzcmip smithi079 running (47s) 28s ago 46s 15.7M - 18.2.1 5be31c24972a bff0ea9e7478 2024-05-04T12:15:57.536 INFO:teuthology.orchestra.run.smithi079.stdout:mds.cephfs.smithi079.pspich smithi079 running (54s) 28s ago 53s 11.4M - 18.2.1 5be31c24972a a01f5acecbb0 2024-05-04T12:15:57.536 INFO:teuthology.orchestra.run.smithi079.stdout:mds.cephfs.smithi146.mkcmzm smithi146 running (44s) 33s ago 43s 15.4M - 18.2.1 5be31c24972a b3cb82b947d5 2024-05-04T12:15:57.536 INFO:teuthology.orchestra.run.smithi079.stdout:mds.cephfs.smithi146.tfajce smithi146 running (50s) 33s ago 50s 12.2M - 18.2.1 5be31c24972a 83ba8a54f68c 2024-05-04T12:15:57.536 INFO:teuthology.orchestra.run.smithi079.stdout:mgr.smithi079.dgcpwl smithi079 *:9283,8765,8443 running (7m) 28s ago 7m 488M - 18.2.1 5be31c24972a 21dc78f7be3a 2024-05-04T12:15:57.536 INFO:teuthology.orchestra.run.smithi079.stdout:mgr.smithi146.eubqih smithi146 *:8443,9283,8765 running (5m) 33s ago 5m 440M - 18.2.1 5be31c24972a e03895ea0bc8 2024-05-04T12:15:57.536 INFO:teuthology.orchestra.run.smithi079.stdout:mon.smithi079 smithi079 running (8m) 28s ago 8m 46.0M 2048M 18.2.1 5be31c24972a ce54f11f45df 2024-05-04T12:15:57.536 INFO:teuthology.orchestra.run.smithi079.stdout:mon.smithi146 smithi146 running (5m) 33s ago 5m 36.3M 2048M 18.2.1 5be31c24972a 085b6b3257ee 2024-05-04T12:15:57.537 INFO:teuthology.orchestra.run.smithi079.stdout:node-exporter.smithi079 smithi079 *:9100 running (6m) 28s ago 6m 15.8M - 1.5.0 0da6a335fe13 3502f0e86c17 2024-05-04T12:15:57.537 INFO:teuthology.orchestra.run.smithi079.stdout:node-exporter.smithi146 smithi146 *:9100 running (5m) 33s ago 5m 17.5M - 1.5.0 0da6a335fe13 19d4f0087b1f 2024-05-04T12:15:57.537 INFO:teuthology.orchestra.run.smithi079.stdout:osd.0 smithi079 running (4m) 28s ago 4m 49.9M 3946M 18.2.1 5be31c24972a ee518ff3d765 2024-05-04T12:15:57.537 INFO:teuthology.orchestra.run.smithi079.stdout:osd.1 smithi079 running (4m) 28s ago 4m 47.0M 3946M 18.2.1 5be31c24972a 57cfd8dc67dd 2024-05-04T12:15:57.537 INFO:teuthology.orchestra.run.smithi079.stdout:osd.2 smithi079 running (3m) 28s ago 3m 46.7M 3946M 18.2.1 5be31c24972a 4778a780462b 2024-05-04T12:15:57.537 INFO:teuthology.orchestra.run.smithi079.stdout:osd.3 smithi146 running (3m) 33s ago 3m 46.1M 4970M 18.2.1 5be31c24972a 1739155eb370 2024-05-04T12:15:57.537 INFO:teuthology.orchestra.run.smithi079.stdout:osd.4 smithi146 running (2m) 33s ago 2m 45.4M 4970M 18.2.1 5be31c24972a be73d59d6afe 2024-05-04T12:15:57.537 INFO:teuthology.orchestra.run.smithi079.stdout:osd.5 smithi146 running (2m) 33s ago 2m 46.3M 4970M 18.2.1 5be31c24972a ce30a9236f6c 2024-05-04T12:15:57.537 INFO:teuthology.orchestra.run.smithi079.stdout:prometheus.smithi079 smithi079 *:9095 running (4m) 28s ago 5m 57.9M - 2.43.0 a07b618ecd1d 1b5e4dd69692 2024-05-04T12:15:57.538 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:57.538+0000 7f161e27b700 1 -- 172.21.15.79:0/2791341803 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f160006c4e0 msgr2=0x7f160006e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:15:57.538 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:57.538+0000 7f161e27b700 1 --2- 172.21.15.79:0/2791341803 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f160006c4e0 0x7f160006e9a0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f160c005fd0 tx=0x7f160c005f60 comp rx=0 tx=0).stop 2024-05-04T12:15:57.538 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:57.538+0000 7f161e27b700 1 -- 172.21.15.79:0/2791341803 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f16180fed20 msgr2=0x7f1618192800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:15:57.538 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:57.538+0000 7f161e27b700 1 --2- 172.21.15.79:0/2791341803 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f16180fed20 0x7f1618192800 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f160800bda0 tx=0x7f160800be80 comp rx=0 tx=0).stop 2024-05-04T12:15:57.538 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:57.538+0000 7f161e27b700 1 -- 172.21.15.79:0/2791341803 shutdown_connections 2024-05-04T12:15:57.538 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:57.538+0000 7f161e27b700 1 --2- 172.21.15.79:0/2791341803 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f160006c4e0 0x7f160006e9a0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:57.538 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:57.538+0000 7f161e27b700 1 --2- 172.21.15.79:0/2791341803 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f16180fed20 0x7f1618192800 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:57.538 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:57.538+0000 7f161e27b700 1 --2- 172.21.15.79:0/2791341803 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f16181022a0 0x7f1618192d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:15:57.538 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:57.538+0000 7f161e27b700 1 -- 172.21.15.79:0/2791341803 >> 172.21.15.79:0/2791341803 conn(0x7f16180fa8c0 msgr2=0x7f1618103260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:15:57.538 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:57.538+0000 7f161e27b700 1 -- 172.21.15.79:0/2791341803 shutdown_connections 2024-05-04T12:15:57.538 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:15:57.538+0000 7f161e27b700 1 -- 172.21.15.79:0/2791341803 wait complete. 2024-05-04T12:15:58.229 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d9b985c4-0a0e-11ef-bc97-c7b262605968 -e sha1=09dbd6bba52d19ba471c33bc15c008e6ad158ea6 -- bash -c 'ceph orch upgrade status' 2024-05-04T12:15:58.550 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/d9b985c4-0a0e-11ef-bc97-c7b262605968/mon.smithi079/config 2024-05-04T12:15:59.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:59 smithi079 ceph-mon[27222]: from='client.14566 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-04T12:15:59.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:15:59 smithi079 ceph-mon[27222]: pgmap v208: 65 pgs: 65 active+clean; 584 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-04T12:15:59.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:59 smithi146 ceph-mon[35311]: from='client.14566 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-04T12:15:59.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:15:59 smithi146 ceph-mon[35311]: pgmap v208: 65 pgs: 65 active+clean; 584 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-04T12:16:00.456 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:00.456+0000 7fdeedda3700 1 -- 172.21.15.79:0/3214640399 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdee8101150 msgr2=0x7fdee8101570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:16:00.456 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:00.456+0000 7fdeedda3700 1 --2- 172.21.15.79:0/3214640399 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdee8101150 0x7fdee8101570 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fded8009910 tx=0x7fded8009c20 comp rx=0 tx=0).stop 2024-05-04T12:16:00.456 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:00.456+0000 7fdeedda3700 1 -- 172.21.15.79:0/3214640399 shutdown_connections 2024-05-04T12:16:00.457 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:00.456+0000 7fdeedda3700 1 --2- 172.21.15.79:0/3214640399 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdee80ffdb0 0x7fdee8100230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:16:00.457 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:00.456+0000 7fdeedda3700 1 --2- 172.21.15.79:0/3214640399 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdee8101150 0x7fdee8101570 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:16:00.457 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:00.456+0000 7fdeedda3700 1 -- 172.21.15.79:0/3214640399 >> 172.21.15.79:0/3214640399 conn(0x7fdee80fa8c0 msgr2=0x7fdee80fcd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:16:00.457 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:00.456+0000 7fdeedda3700 1 -- 172.21.15.79:0/3214640399 shutdown_connections 2024-05-04T12:16:00.457 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:00.456+0000 7fdeedda3700 1 -- 172.21.15.79:0/3214640399 wait complete. 2024-05-04T12:16:00.458 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:00.458+0000 7fdeedda3700 1 Processor -- start 2024-05-04T12:16:00.459 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:00.458+0000 7fdeedda3700 1 -- start start 2024-05-04T12:16:00.459 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:00.458+0000 7fdeedda3700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdee80ffdb0 0x7fdee81953c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:16:00.459 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:00.458+0000 7fdeedda3700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdee8101150 0x7fdee8195900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:16:00.459 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:00.458+0000 7fdeedda3700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdee8195f20 con 0x7fdee8101150 2024-05-04T12:16:00.459 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:00.458+0000 7fdeedda3700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdee8196060 con 0x7fdee80ffdb0 2024-05-04T12:16:00.459 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:00.459+0000 7fdee77fe700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdee80ffdb0 0x7fdee81953c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:16:00.459 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:00.459+0000 7fdee6ffd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdee8101150 0x7fdee8195900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:16:00.459 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:00.459+0000 7fdee6ffd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdee8101150 0x7fdee8195900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:58902/0 (socket says 172.21.15.79:58902) 2024-05-04T12:16:00.459 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:00.459+0000 7fdee6ffd700 1 -- 172.21.15.79:0/2492689934 learned_addr learned my addr 172.21.15.79:0/2492689934 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:16:00.459 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:00.459+0000 7fdee6ffd700 1 -- 172.21.15.79:0/2492689934 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdee80ffdb0 msgr2=0x7fdee81953c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:16:00.459 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:00.459+0000 7fdee6ffd700 1 --2- 172.21.15.79:0/2492689934 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdee80ffdb0 0x7fdee81953c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:16:00.460 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:00.459+0000 7fdee6ffd700 1 -- 172.21.15.79:0/2492689934 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fded80095f0 con 0x7fdee8101150 2024-05-04T12:16:00.460 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:00.459+0000 7fdee6ffd700 1 --2- 172.21.15.79:0/2492689934 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdee8101150 0x7fdee8195900 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fdedc00eab0 tx=0x7fdedc00edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:16:00.461 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:00.460+0000 7fdee4ff9700 1 -- 172.21.15.79:0/2492689934 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdedc013070 con 0x7fdee8101150 2024-05-04T12:16:00.461 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:00.460+0000 7fdee4ff9700 1 -- 172.21.15.79:0/2492689934 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fdedc017410 con 0x7fdee8101150 2024-05-04T12:16:00.461 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:00.460+0000 7fdeedda3700 1 -- 172.21.15.79:0/2492689934 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdee80fed00 con 0x7fdee8101150 2024-05-04T12:16:00.461 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:00.460+0000 7fdeedda3700 1 -- 172.21.15.79:0/2492689934 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdee80ff250 con 0x7fdee8101150 2024-05-04T12:16:00.461 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:00.460+0000 7fdee4ff9700 1 -- 172.21.15.79:0/2492689934 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdedc0165e0 con 0x7fdee8101150 2024-05-04T12:16:00.461 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:00.461+0000 7fdeedda3700 1 -- 172.21.15.79:0/2492689934 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fded4005320 con 0x7fdee8101150 2024-05-04T12:16:00.461 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:00.462+0000 7fdee4ff9700 1 -- 172.21.15.79:0/2492689934 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7fdedc00bb10 con 0x7fdee8101150 2024-05-04T12:16:00.463 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:00.463+0000 7fdee4ff9700 1 --2- 172.21.15.79:0/2492689934 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7fded006c4e0 0x7fded006e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:16:00.463 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:00.463+0000 7fdee4ff9700 1 -- 172.21.15.79:0/2492689934 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fdedc08d1c0 con 0x7fdee8101150 2024-05-04T12:16:00.463 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:00.463+0000 7fdee77fe700 1 --2- 172.21.15.79:0/2492689934 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7fded006c4e0 0x7fded006e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:16:00.463 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:00.464+0000 7fdee77fe700 1 --2- 172.21.15.79:0/2492689934 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7fded006c4e0 0x7fded006e9a0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fded8009fd0 tx=0x7fded8005940 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:16:00.471 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:00.471+0000 7fdee4ff9700 1 -- 172.21.15.79:0/2492689934 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fdedc05b3b0 con 0x7fdee8101150 2024-05-04T12:16:00.900 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:00.900+0000 7fdeedda3700 1 -- 172.21.15.79:0/2492689934 --> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fded4000bf0 con 0x7fded006c4e0 2024-05-04T12:16:00.901 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:00.902+0000 7fdee4ff9700 1 -- 172.21.15.79:0/2492689934 <== mgr.14219 v2:172.21.15.79:6800/402764217 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+329 (secure 0 0 0) 0x7fded4000bf0 con 0x7fded006c4e0 2024-05-04T12:16:00.902 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-05-04T12:16:00.902 INFO:teuthology.orchestra.run.smithi079.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:09dbd6bba52d19ba471c33bc15c008e6ad158ea6", 2024-05-04T12:16:00.902 INFO:teuthology.orchestra.run.smithi079.stdout: "in_progress": true, 2024-05-04T12:16:00.902 INFO:teuthology.orchestra.run.smithi079.stdout: "which": "Upgrading all daemon types on all hosts", 2024-05-04T12:16:00.902 INFO:teuthology.orchestra.run.smithi079.stdout: "services_complete": [], 2024-05-04T12:16:00.902 INFO:teuthology.orchestra.run.smithi079.stdout: "progress": "", 2024-05-04T12:16:00.902 INFO:teuthology.orchestra.run.smithi079.stdout: "message": "Error: UPGRADE_FAILED_PULL: Upgrade: failed to pull target image", 2024-05-04T12:16:00.902 INFO:teuthology.orchestra.run.smithi079.stdout: "is_paused": true 2024-05-04T12:16:00.902 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-05-04T12:16:00.905 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:00.905+0000 7fdeedda3700 1 -- 172.21.15.79:0/2492689934 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7fded006c4e0 msgr2=0x7fded006e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:16:00.905 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:00.905+0000 7fdeedda3700 1 --2- 172.21.15.79:0/2492689934 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7fded006c4e0 0x7fded006e9a0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fded8009fd0 tx=0x7fded8005940 comp rx=0 tx=0).stop 2024-05-04T12:16:00.905 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:00.905+0000 7fdeedda3700 1 -- 172.21.15.79:0/2492689934 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdee8101150 msgr2=0x7fdee8195900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:16:00.905 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:00.905+0000 7fdeedda3700 1 --2- 172.21.15.79:0/2492689934 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdee8101150 0x7fdee8195900 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fdedc00eab0 tx=0x7fdedc00edc0 comp rx=0 tx=0).stop 2024-05-04T12:16:00.905 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:00.905+0000 7fdeedda3700 1 -- 172.21.15.79:0/2492689934 shutdown_connections 2024-05-04T12:16:00.905 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:00.905+0000 7fdeedda3700 1 --2- 172.21.15.79:0/2492689934 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7fded006c4e0 0x7fded006e9a0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:16:00.905 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:00.905+0000 7fdeedda3700 1 --2- 172.21.15.79:0/2492689934 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdee80ffdb0 0x7fdee81953c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:16:00.905 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:00.905+0000 7fdeedda3700 1 --2- 172.21.15.79:0/2492689934 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdee8101150 0x7fdee8195900 secure :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fdedc00eab0 tx=0x7fdedc00edc0 comp rx=0 tx=0).stop 2024-05-04T12:16:00.905 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:00.905+0000 7fdeedda3700 1 -- 172.21.15.79:0/2492689934 >> 172.21.15.79:0/2492689934 conn(0x7fdee80fa8c0 msgr2=0x7fdee8103ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:16:00.905 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:00.905+0000 7fdeedda3700 1 -- 172.21.15.79:0/2492689934 shutdown_connections 2024-05-04T12:16:00.905 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:00.906+0000 7fdeedda3700 1 -- 172.21.15.79:0/2492689934 wait complete. 2024-05-04T12:16:01.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:16:01 smithi079 ceph-mon[27222]: pgmap v209: 65 pgs: 65 active+clean; 584 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-04T12:16:01.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:16:01 smithi146 ceph-mon[35311]: pgmap v209: 65 pgs: 65 active+clean; 584 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-04T12:16:01.610 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d9b985c4-0a0e-11ef-bc97-c7b262605968 -e sha1=09dbd6bba52d19ba471c33bc15c008e6ad158ea6 -- bash -c 'ceph health detail' 2024-05-04T12:16:01.841 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/d9b985c4-0a0e-11ef-bc97-c7b262605968/mon.smithi079/config 2024-05-04T12:16:02.308 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:16:02 smithi079 ceph-mon[27222]: from='client.14570 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-04T12:16:02.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:16:02 smithi146 ceph-mon[35311]: from='client.14570 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-04T12:16:03.212 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:16:03 smithi079 ceph-mon[27222]: pgmap v210: 65 pgs: 65 active+clean; 584 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-04T12:16:03.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:16:03 smithi146 ceph-mon[35311]: pgmap v210: 65 pgs: 65 active+clean; 584 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-04T12:16:03.596 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:03.596+0000 7fc94f55a700 1 -- 172.21.15.79:0/2233933793 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc948100dc0 msgr2=0x7fc948101220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:16:03.596 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:03.596+0000 7fc94f55a700 1 --2- 172.21.15.79:0/2233933793 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc948100dc0 0x7fc948101220 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fc938009910 tx=0x7fc938009c20 comp rx=0 tx=0).stop 2024-05-04T12:16:03.596 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:03.596+0000 7fc94f55a700 1 -- 172.21.15.79:0/2233933793 shutdown_connections 2024-05-04T12:16:03.596 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:03.596+0000 7fc94f55a700 1 --2- 172.21.15.79:0/2233933793 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc948100dc0 0x7fc948101220 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:16:03.596 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:03.596+0000 7fc94f55a700 1 --2- 172.21.15.79:0/2233933793 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc9480ffbc0 0x7fc9480fffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:16:03.596 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:03.596+0000 7fc94f55a700 1 -- 172.21.15.79:0/2233933793 >> 172.21.15.79:0/2233933793 conn(0x7fc9480fb160 msgr2=0x7fc9480fd5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:16:03.596 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:03.596+0000 7fc94f55a700 1 -- 172.21.15.79:0/2233933793 shutdown_connections 2024-05-04T12:16:03.596 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:03.596+0000 7fc94f55a700 1 -- 172.21.15.79:0/2233933793 wait complete. 2024-05-04T12:16:03.598 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:03.597+0000 7fc94f55a700 1 Processor -- start 2024-05-04T12:16:03.598 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:03.597+0000 7fc94f55a700 1 -- start start 2024-05-04T12:16:03.598 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:03.597+0000 7fc94f55a700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc9480ffbc0 0x7fc9481956c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:16:03.598 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:03.597+0000 7fc94f55a700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc948100dc0 0x7fc948195c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:16:03.598 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:03.598+0000 7fc94f55a700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc948196220 con 0x7fc9480ffbc0 2024-05-04T12:16:03.598 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:03.598+0000 7fc94f55a700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc948196360 con 0x7fc948100dc0 2024-05-04T12:16:03.598 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:03.598+0000 7fc94d2f6700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc9480ffbc0 0x7fc9481956c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:16:03.598 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:03.598+0000 7fc94caf5700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc948100dc0 0x7fc948195c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:16:03.598 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:03.598+0000 7fc94caf5700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc948100dc0 0x7fc948195c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.79:55094/0 (socket says 172.21.15.79:55094) 2024-05-04T12:16:03.598 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:03.598+0000 7fc94caf5700 1 -- 172.21.15.79:0/2432885106 learned_addr learned my addr 172.21.15.79:0/2432885106 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:16:03.598 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:03.598+0000 7fc94d2f6700 1 -- 172.21.15.79:0/2432885106 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc948100dc0 msgr2=0x7fc948195c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:16:03.598 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:03.598+0000 7fc94d2f6700 1 --2- 172.21.15.79:0/2432885106 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc948100dc0 0x7fc948195c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:16:03.598 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:03.598+0000 7fc94d2f6700 1 -- 172.21.15.79:0/2432885106 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9380095f0 con 0x7fc9480ffbc0 2024-05-04T12:16:03.599 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:03.598+0000 7fc94d2f6700 1 --2- 172.21.15.79:0/2432885106 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc9480ffbc0 0x7fc9481956c0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fc94400d6c0 tx=0x7fc94400d9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:16:03.599 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:03.599+0000 7fc93e7fc700 1 -- 172.21.15.79:0/2432885106 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc9440041f0 con 0x7fc9480ffbc0 2024-05-04T12:16:03.599 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:03.599+0000 7fc93e7fc700 1 -- 172.21.15.79:0/2432885106 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fc944004350 con 0x7fc9480ffbc0 2024-05-04T12:16:03.599 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:03.599+0000 7fc93e7fc700 1 -- 172.21.15.79:0/2432885106 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc944003d20 con 0x7fc9480ffbc0 2024-05-04T12:16:03.599 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:03.599+0000 7fc94f55a700 1 -- 172.21.15.79:0/2432885106 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc94819ae10 con 0x7fc9480ffbc0 2024-05-04T12:16:03.599 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:03.599+0000 7fc94f55a700 1 -- 172.21.15.79:0/2432885106 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc94819b2d0 con 0x7fc9480ffbc0 2024-05-04T12:16:03.600 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:03.600+0000 7fc94f55a700 1 -- 172.21.15.79:0/2432885106 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc94818f530 con 0x7fc9480ffbc0 2024-05-04T12:16:03.600 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:03.601+0000 7fc93e7fc700 1 -- 172.21.15.79:0/2432885106 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7fc94400f690 con 0x7fc9480ffbc0 2024-05-04T12:16:03.601 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:03.601+0000 7fc93e7fc700 1 --2- 172.21.15.79:0/2432885106 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7fc93406c530 0x7fc93406e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:16:03.601 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:03.601+0000 7fc93e7fc700 1 -- 172.21.15.79:0/2432885106 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fc94408b760 con 0x7fc9480ffbc0 2024-05-04T12:16:03.601 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:03.602+0000 7fc94caf5700 1 --2- 172.21.15.79:0/2432885106 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7fc93406c530 0x7fc93406e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:16:03.602 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:03.602+0000 7fc94caf5700 1 --2- 172.21.15.79:0/2432885106 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7fc93406c530 0x7fc93406e9f0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fc938006010 tx=0x7fc93800b540 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:16:03.609 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:03.609+0000 7fc93e7fc700 1 -- 172.21.15.79:0/2432885106 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fc9440599d0 con 0x7fc9480ffbc0 2024-05-04T12:16:03.900 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:03.900+0000 7fc94f55a700 1 -- 172.21.15.79:0/2432885106 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fc948068af0 con 0x7fc9480ffbc0 2024-05-04T12:16:03.900 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:03.901+0000 7fc93e7fc700 1 -- 172.21.15.79:0/2432885106 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+1054 (secure 0 0 0) 0x7fc944016430 con 0x7fc9480ffbc0 2024-05-04T12:16:03.901 INFO:teuthology.orchestra.run.smithi079.stdout:HEALTH_WARN 1 filesystem with deprecated feature inline_data; Upgrade: failed to pull target image 2024-05-04T12:16:03.901 INFO:teuthology.orchestra.run.smithi079.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-05-04T12:16:03.901 INFO:teuthology.orchestra.run.smithi079.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-05-04T12:16:03.901 INFO:teuthology.orchestra.run.smithi079.stdout:[WRN] UPGRADE_FAILED_PULL: Upgrade: failed to pull target image 2024-05-04T12:16:03.901 INFO:teuthology.orchestra.run.smithi079.stdout: host smithi079 `cephadm pull` failed: cephadm exited with an error code: 1, stderr: Pulling container image quay.ceph.io/ceph-ci/ceph:09dbd6bba52d19ba471c33bc15c008e6ad158ea6... 2024-05-04T12:16:03.901 INFO:teuthology.orchestra.run.smithi079.stdout:Non-zero exit code 125 from /usr/bin/podman pull quay.ceph.io/ceph-ci/ceph:09dbd6bba52d19ba471c33bc15c008e6ad158ea6 2024-05-04T12:16:03.902 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/podman: stderr Trying to pull quay.ceph.io/ceph-ci/ceph:09dbd6bba52d19ba471c33bc15c008e6ad158ea6... 2024-05-04T12:16:03.902 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/podman: stderr Error: initializing source docker://quay.ceph.io/ceph-ci/ceph:09dbd6bba52d19ba471c33bc15c008e6ad158ea6: reading manifest 09dbd6bba52d19ba471c33bc15c008e6ad158ea6 in quay.ceph.io/ceph-ci/ceph: manifest unknown 2024-05-04T12:16:03.902 INFO:teuthology.orchestra.run.smithi079.stdout:ERROR: Failed command: /usr/bin/podman pull quay.ceph.io/ceph-ci/ceph:09dbd6bba52d19ba471c33bc15c008e6ad158ea6 2024-05-04T12:16:03.903 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:03.903+0000 7fc94f55a700 1 -- 172.21.15.79:0/2432885106 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7fc93406c530 msgr2=0x7fc93406e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:16:03.903 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:03.903+0000 7fc94f55a700 1 --2- 172.21.15.79:0/2432885106 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7fc93406c530 0x7fc93406e9f0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fc938006010 tx=0x7fc93800b540 comp rx=0 tx=0).stop 2024-05-04T12:16:03.904 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:03.903+0000 7fc94f55a700 1 -- 172.21.15.79:0/2432885106 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc9480ffbc0 msgr2=0x7fc9481956c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:16:03.904 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:03.903+0000 7fc94f55a700 1 --2- 172.21.15.79:0/2432885106 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc9480ffbc0 0x7fc9481956c0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fc94400d6c0 tx=0x7fc94400d9d0 comp rx=0 tx=0).stop 2024-05-04T12:16:03.904 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:03.903+0000 7fc94f55a700 1 -- 172.21.15.79:0/2432885106 shutdown_connections 2024-05-04T12:16:03.904 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:03.903+0000 7fc94f55a700 1 --2- 172.21.15.79:0/2432885106 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7fc93406c530 0x7fc93406e9f0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:16:03.904 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:03.903+0000 7fc94f55a700 1 --2- 172.21.15.79:0/2432885106 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc9480ffbc0 0x7fc9481956c0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:16:03.904 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:03.903+0000 7fc94f55a700 1 --2- 172.21.15.79:0/2432885106 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc948100dc0 0x7fc948195c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:16:03.905 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:03.903+0000 7fc94f55a700 1 -- 172.21.15.79:0/2432885106 >> 172.21.15.79:0/2432885106 conn(0x7fc9480fb160 msgr2=0x7fc948104110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:16:03.905 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:03.904+0000 7fc94f55a700 1 -- 172.21.15.79:0/2432885106 shutdown_connections 2024-05-04T12:16:03.905 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:03.904+0000 7fc94f55a700 1 -- 172.21.15.79:0/2432885106 wait complete. 2024-05-04T12:16:04.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:16:04 smithi079 ceph-mon[27222]: from='client.? 172.21.15.79:0/2432885106' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-04T12:16:04.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:16:04 smithi146 ceph-mon[35311]: from='client.? 172.21.15.79:0/2432885106' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-04T12:16:05.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:16:05 smithi079 ceph-mon[27222]: pgmap v211: 65 pgs: 65 active+clean; 584 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-04T12:16:05.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:16:05 smithi146 ceph-mon[35311]: pgmap v211: 65 pgs: 65 active+clean; 584 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-04T12:16:07.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:16:07 smithi079 ceph-mon[27222]: pgmap v212: 65 pgs: 65 active+clean; 584 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-04T12:16:07.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:16:07 smithi146 ceph-mon[35311]: pgmap v212: 65 pgs: 65 active+clean; 584 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-04T12:16:07.817 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d9b985c4-0a0e-11ef-bc97-c7b262605968 -e sha1=09dbd6bba52d19ba471c33bc15c008e6ad158ea6 -- bash -c 'ceph versions' 2024-05-04T12:16:08.032 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/d9b985c4-0a0e-11ef-bc97-c7b262605968/mon.smithi079/config 2024-05-04T12:16:09.397 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:16:09 smithi079 ceph-mon[27222]: pgmap v213: 65 pgs: 65 active+clean; 584 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-04T12:16:09.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:16:09 smithi146 ceph-mon[35311]: pgmap v213: 65 pgs: 65 active+clean; 584 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-04T12:16:09.883 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:09.882+0000 7f9e04e53700 1 -- 172.21.15.79:0/1562113283 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9e00104130 msgr2=0x7f9e00106520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:16:09.883 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:09.882+0000 7f9e04e53700 1 --2- 172.21.15.79:0/1562113283 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9e00104130 0x7f9e00106520 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f9df4009910 tx=0x7f9df4009c20 comp rx=0 tx=0).stop 2024-05-04T12:16:09.883 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:09.883+0000 7f9e04e53700 1 -- 172.21.15.79:0/1562113283 shutdown_connections 2024-05-04T12:16:09.883 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:09.883+0000 7f9e04e53700 1 --2- 172.21.15.79:0/1562113283 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9e00104130 0x7f9e00106520 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:16:09.883 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:09.883+0000 7f9e04e53700 1 --2- 172.21.15.79:0/1562113283 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9e00101800 0x7f9e00103bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:16:09.883 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:09.883+0000 7f9e04e53700 1 -- 172.21.15.79:0/1562113283 >> 172.21.15.79:0/1562113283 conn(0x7f9e000fb190 msgr2=0x7f9e000fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:16:09.883 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:09.883+0000 7f9e04e53700 1 -- 172.21.15.79:0/1562113283 shutdown_connections 2024-05-04T12:16:09.883 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:09.883+0000 7f9e04e53700 1 -- 172.21.15.79:0/1562113283 wait complete. 2024-05-04T12:16:09.885 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:09.884+0000 7f9e04e53700 1 Processor -- start 2024-05-04T12:16:09.885 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:09.884+0000 7f9e04e53700 1 -- start start 2024-05-04T12:16:09.885 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:09.884+0000 7f9e04e53700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9e00101800 0x7f9e0019ca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:16:09.885 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:09.884+0000 7f9e04e53700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9e00104130 0x7f9e0019cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:16:09.885 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:09.884+0000 7f9e04e53700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e0019d5f0 con 0x7f9e00104130 2024-05-04T12:16:09.886 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:09.884+0000 7f9e04e53700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e0019d730 con 0x7f9e00101800 2024-05-04T12:16:09.886 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:09.885+0000 7f9dfdd9b700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9e00104130 0x7f9e0019cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:16:09.886 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:09.885+0000 7f9dfdd9b700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9e00104130 0x7f9e0019cfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:40102/0 (socket says 172.21.15.79:40102) 2024-05-04T12:16:09.886 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:09.885+0000 7f9dfdd9b700 1 -- 172.21.15.79:0/2106878417 learned_addr learned my addr 172.21.15.79:0/2106878417 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-05-04T12:16:09.886 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:09.885+0000 7f9dfe59c700 1 --2- 172.21.15.79:0/2106878417 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9e00101800 0x7f9e0019ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:16:09.886 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:09.885+0000 7f9dfdd9b700 1 -- 172.21.15.79:0/2106878417 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9e00101800 msgr2=0x7f9e0019ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:16:09.886 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:09.885+0000 7f9dfdd9b700 1 --2- 172.21.15.79:0/2106878417 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9e00101800 0x7f9e0019ca90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:16:09.886 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:09.885+0000 7f9dfdd9b700 1 -- 172.21.15.79:0/2106878417 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9df40095f0 con 0x7f9e00104130 2024-05-04T12:16:09.886 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:09.885+0000 7f9dfe59c700 1 --2- 172.21.15.79:0/2106878417 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9e00101800 0x7f9e0019ca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-04T12:16:09.886 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:09.885+0000 7f9dfdd9b700 1 --2- 172.21.15.79:0/2106878417 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9e00104130 0x7f9e0019cfd0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f9df4000c00 tx=0x7f9df400f740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:16:09.886 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:09.885+0000 7f9def7fe700 1 -- 172.21.15.79:0/2106878417 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9df401d070 con 0x7f9e00104130 2024-05-04T12:16:09.886 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:09.886+0000 7f9def7fe700 1 -- 172.21.15.79:0/2106878417 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f9df4005dc0 con 0x7f9e00104130 2024-05-04T12:16:09.886 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:09.886+0000 7f9def7fe700 1 -- 172.21.15.79:0/2106878417 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9df40226a0 con 0x7f9e00104130 2024-05-04T12:16:09.886 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:09.886+0000 7f9e04e53700 1 -- 172.21.15.79:0/2106878417 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e001a2180 con 0x7f9e00104130 2024-05-04T12:16:09.887 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:09.886+0000 7f9e04e53700 1 -- 172.21.15.79:0/2106878417 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e001a2670 con 0x7f9e00104130 2024-05-04T12:16:09.887 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:09.887+0000 7f9def7fe700 1 -- 172.21.15.79:0/2106878417 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f9df4022800 con 0x7f9e00104130 2024-05-04T12:16:09.887 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:09.888+0000 7f9e04e53700 1 -- 172.21.15.79:0/2106878417 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e000fcd80 con 0x7f9e00104130 2024-05-04T12:16:09.888 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:09.888+0000 7f9def7fe700 1 --2- 172.21.15.79:0/2106878417 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f9de806c4e0 0x7f9de806e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T12:16:09.888 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:09.888+0000 7f9def7fe700 1 -- 172.21.15.79:0/2106878417 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f9df408ddc0 con 0x7f9e00104130 2024-05-04T12:16:09.888 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:09.888+0000 7f9dfe59c700 1 --2- 172.21.15.79:0/2106878417 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f9de806c4e0 0x7f9de806e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T12:16:09.888 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:09.889+0000 7f9dfe59c700 1 --2- 172.21.15.79:0/2106878417 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f9de806c4e0 0x7f9de806e9a0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f9df0009cb0 tx=0x7f9df0009260 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T12:16:09.896 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:09.897+0000 7f9def7fe700 1 -- 172.21.15.79:0/2106878417 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f9df405c050 con 0x7f9e00104130 2024-05-04T12:16:10.263 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:10.264+0000 7f9e04e53700 1 -- 172.21.15.79:0/2106878417 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f9e0002cc70 con 0x7f9e00104130 2024-05-04T12:16:10.264 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:10.264+0000 7f9def7fe700 1 -- 172.21.15.79:0/2106878417 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+558 (secure 0 0 0) 0x7f9df405bbe0 con 0x7f9e00104130 2024-05-04T12:16:10.264 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-05-04T12:16:10.264 INFO:teuthology.orchestra.run.smithi079.stdout: "mon": { 2024-05-04T12:16:10.264 INFO:teuthology.orchestra.run.smithi079.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 2 2024-05-04T12:16:10.264 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-05-04T12:16:10.264 INFO:teuthology.orchestra.run.smithi079.stdout: "mgr": { 2024-05-04T12:16:10.264 INFO:teuthology.orchestra.run.smithi079.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 2 2024-05-04T12:16:10.264 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-05-04T12:16:10.264 INFO:teuthology.orchestra.run.smithi079.stdout: "osd": { 2024-05-04T12:16:10.264 INFO:teuthology.orchestra.run.smithi079.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 6 2024-05-04T12:16:10.265 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-05-04T12:16:10.265 INFO:teuthology.orchestra.run.smithi079.stdout: "mds": { 2024-05-04T12:16:10.265 INFO:teuthology.orchestra.run.smithi079.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 4 2024-05-04T12:16:10.265 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-05-04T12:16:10.265 INFO:teuthology.orchestra.run.smithi079.stdout: "overall": { 2024-05-04T12:16:10.265 INFO:teuthology.orchestra.run.smithi079.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 14 2024-05-04T12:16:10.265 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-05-04T12:16:10.265 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-05-04T12:16:10.267 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:10.267+0000 7f9e04e53700 1 -- 172.21.15.79:0/2106878417 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f9de806c4e0 msgr2=0x7f9de806e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:16:10.267 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:10.267+0000 7f9e04e53700 1 --2- 172.21.15.79:0/2106878417 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f9de806c4e0 0x7f9de806e9a0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f9df0009cb0 tx=0x7f9df0009260 comp rx=0 tx=0).stop 2024-05-04T12:16:10.267 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:10.267+0000 7f9e04e53700 1 -- 172.21.15.79:0/2106878417 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9e00104130 msgr2=0x7f9e0019cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T12:16:10.267 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:10.267+0000 7f9e04e53700 1 --2- 172.21.15.79:0/2106878417 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9e00104130 0x7f9e0019cfd0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f9df4000c00 tx=0x7f9df400f740 comp rx=0 tx=0).stop 2024-05-04T12:16:10.267 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:10.267+0000 7f9e04e53700 1 -- 172.21.15.79:0/2106878417 shutdown_connections 2024-05-04T12:16:10.267 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:10.267+0000 7f9e04e53700 1 --2- 172.21.15.79:0/2106878417 >> [v2:172.21.15.79:6800/402764217,v1:172.21.15.79:6801/402764217] conn(0x7f9de806c4e0 0x7f9de806e9a0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:16:10.267 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:10.267+0000 7f9e04e53700 1 --2- 172.21.15.79:0/2106878417 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9e00101800 0x7f9e0019ca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:16:10.267 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:10.267+0000 7f9e04e53700 1 --2- 172.21.15.79:0/2106878417 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9e00104130 0x7f9e0019cfd0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T12:16:10.267 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:10.267+0000 7f9e04e53700 1 -- 172.21.15.79:0/2106878417 >> 172.21.15.79:0/2106878417 conn(0x7f9e000fb190 msgr2=0x7f9e00104da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T12:16:10.267 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:10.267+0000 7f9e04e53700 1 -- 172.21.15.79:0/2106878417 shutdown_connections 2024-05-04T12:16:10.267 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:16:10.267+0000 7f9e04e53700 1 -- 172.21.15.79:0/2106878417 wait complete. 2024-05-04T12:16:10.941 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d9b985c4-0a0e-11ef-bc97-c7b262605968 -e sha1=09dbd6bba52d19ba471c33bc15c008e6ad158ea6 -- bash -c 'echo "wait for servicemap items w/ changing names to refresh"' 2024-05-04T12:16:11.167 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/d9b985c4-0a0e-11ef-bc97-c7b262605968/mon.smithi079/config 2024-05-04T12:16:11.195 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:16:11 smithi079 ceph-mon[27222]: pgmap v214: 65 pgs: 65 active+clean; 584 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-04T12:16:11.195 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:16:11 smithi079 ceph-mon[27222]: from='client.? 172.21.15.79:0/2106878417' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-04T12:16:11.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:16:11 smithi146 ceph-mon[35311]: pgmap v214: 65 pgs: 65 active+clean; 584 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-04T12:16:11.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:16:11 smithi146 ceph-mon[35311]: from='client.? 172.21.15.79:0/2106878417' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-04T12:16:12.428 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:16:12 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-04T12:16:12.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:16:12 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-04T12:16:12.873 INFO:teuthology.orchestra.run.smithi079.stdout:wait for servicemap items w/ changing names to refresh 2024-05-04T12:16:13.078 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:16:13 smithi079 ceph-mon[27222]: pgmap v215: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 255 B/s wr, 0 op/s 2024-05-04T12:16:13.370 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:16:13 smithi146 ceph-mon[35311]: pgmap v215: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 255 B/s wr, 0 op/s 2024-05-04T12:16:13.617 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d9b985c4-0a0e-11ef-bc97-c7b262605968 -e sha1=09dbd6bba52d19ba471c33bc15c008e6ad158ea6 -- bash -c 'sleep 60' 2024-05-04T12:16:13.883 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/d9b985c4-0a0e-11ef-bc97-c7b262605968/mon.smithi079/config 2024-05-04T12:16:15.373 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:16:15 smithi079 ceph-mon[27222]: pgmap v216: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 341 B/s wr, 0 op/s 2024-05-04T12:16:15.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:16:15 smithi146 ceph-mon[35311]: pgmap v216: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 341 B/s wr, 0 op/s 2024-05-04T12:16:17.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:16:17 smithi079 ceph-mon[27222]: pgmap v217: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 341 B/s wr, 0 op/s 2024-05-04T12:16:17.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:16:17 smithi146 ceph-mon[35311]: pgmap v217: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 341 B/s wr, 0 op/s 2024-05-04T12:16:19.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:16:19 smithi079 ceph-mon[27222]: pgmap v218: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 597 B/s wr, 0 op/s 2024-05-04T12:16:19.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:16:19 smithi146 ceph-mon[35311]: pgmap v218: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 597 B/s wr, 0 op/s 2024-05-04T12:16:21.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:16:21 smithi079 ceph-mon[27222]: pgmap v219: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 597 B/s wr, 0 op/s 2024-05-04T12:16:21.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:16:21 smithi146 ceph-mon[35311]: pgmap v219: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 597 B/s wr, 0 op/s 2024-05-04T12:16:23.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:16:23 smithi079 ceph-mon[27222]: pgmap v220: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 597 B/s wr, 0 op/s 2024-05-04T12:16:23.500 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:16:23 smithi146 ceph-mon[35311]: pgmap v220: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 597 B/s wr, 0 op/s 2024-05-04T12:16:23.869 INFO:tasks.workunit.client.1.smithi146.stderr:Note: switching to '09dbd6bba52d19ba471c33bc15c008e6ad158ea6'. 2024-05-04T12:16:23.869 INFO:tasks.workunit.client.1.smithi146.stderr: 2024-05-04T12:16:23.869 INFO:tasks.workunit.client.1.smithi146.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-05-04T12:16:23.869 INFO:tasks.workunit.client.1.smithi146.stderr:changes and commit them, and you can discard any commits you make in this 2024-05-04T12:16:23.869 INFO:tasks.workunit.client.1.smithi146.stderr:state without impacting any branches by switching back to a branch. 2024-05-04T12:16:23.869 INFO:tasks.workunit.client.1.smithi146.stderr: 2024-05-04T12:16:23.869 INFO:tasks.workunit.client.1.smithi146.stderr:If you want to create a new branch to retain commits you create, you may 2024-05-04T12:16:23.869 INFO:tasks.workunit.client.1.smithi146.stderr:do so (now or later) by using -c with the switch command. Example: 2024-05-04T12:16:23.869 INFO:tasks.workunit.client.1.smithi146.stderr: 2024-05-04T12:16:23.869 INFO:tasks.workunit.client.1.smithi146.stderr: git switch -c 2024-05-04T12:16:23.869 INFO:tasks.workunit.client.1.smithi146.stderr: 2024-05-04T12:16:23.869 INFO:tasks.workunit.client.1.smithi146.stderr:Or undo this operation with: 2024-05-04T12:16:23.870 INFO:tasks.workunit.client.1.smithi146.stderr: 2024-05-04T12:16:23.870 INFO:tasks.workunit.client.1.smithi146.stderr: git switch - 2024-05-04T12:16:23.870 INFO:tasks.workunit.client.1.smithi146.stderr: 2024-05-04T12:16:23.870 INFO:tasks.workunit.client.1.smithi146.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-05-04T12:16:23.870 INFO:tasks.workunit.client.1.smithi146.stderr: 2024-05-04T12:16:23.870 INFO:tasks.workunit.client.1.smithi146.stderr:HEAD is now at 09dbd6bba52 Merge branch 'wip-65150-squid' of https://github.com/sseshasa/ceph into wip-yuri4-testing-2024-04-29-0642 2024-05-04T12:16:23.875 DEBUG:teuthology.orchestra.run.smithi146:> cd -- /home/ubuntu/cephtest/clone.client.1/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.1 2024-05-04T12:16:23.937 INFO:tasks.workunit.client.1.smithi146.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-05-04T12:16:23.939 INFO:tasks.workunit.client.1.smithi146.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/direct_io' 2024-05-04T12:16:23.939 INFO:tasks.workunit.client.1.smithi146.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-05-04T12:16:24.528 INFO:tasks.workunit.client.1.smithi146.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-05-04T12:16:24.583 INFO:tasks.workunit.client.1.smithi146.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-05-04T12:16:24.623 INFO:tasks.workunit.client.1.smithi146.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/direct_io' 2024-05-04T12:16:24.625 INFO:tasks.workunit.client.1.smithi146.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/fs' 2024-05-04T12:16:24.625 INFO:tasks.workunit.client.1.smithi146.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-05-04T12:16:24.663 INFO:tasks.workunit.client.1.smithi146.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/fs' 2024-05-04T12:16:24.666 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-05-04T12:16:24.667 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/home/ubuntu/cephtest/workunits.list.client.1 of=/dev/stdout 2024-05-04T12:16:24.722 INFO:tasks.workunit:Running workunits matching suites/fsstress.sh on client.1... 2024-05-04T12:16:24.726 INFO:tasks.workunit:Running workunit suites/fsstress.sh... 2024-05-04T12:16:24.726 DEBUG:teuthology.orchestra.run.smithi146:workunit test suites/fsstress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=09dbd6bba52d19ba471c33bc15c008e6ad158ea6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/fsstress.sh 2024-05-04T12:16:24.784 INFO:tasks.workunit.client.1.smithi146.stderr:+ mkdir -p fsstress 2024-05-04T12:16:24.785 INFO:tasks.workunit.client.1.smithi146.stderr:+ pushd fsstress 2024-05-04T12:16:24.786 INFO:tasks.workunit.client.1.smithi146.stdout:~/cephtest/mnt.1/client.1/tmp/fsstress ~/cephtest/mnt.1/client.1/tmp 2024-05-04T12:16:24.786 INFO:tasks.workunit.client.1.smithi146.stderr:+ wget -q -O ltp-full.tgz http://download.ceph.com/qa/ltp-full-20091231.tgz 2024-05-04T12:16:25.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:16:25 smithi079 ceph-mon[27222]: pgmap v221: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 426 B/s wr, 0 op/s 2024-05-04T12:16:25.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:16:25 smithi146 ceph-mon[35311]: pgmap v221: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 426 B/s wr, 0 op/s 2024-05-04T12:16:25.692 INFO:tasks.workunit.client.1.smithi146.stderr:+ tar xzf ltp-full.tgz 2024-05-04T12:16:27.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:16:27 smithi079 ceph-mon[27222]: pgmap v222: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 341 B/s wr, 0 op/s 2024-05-04T12:16:27.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:16:27 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-04T12:16:27.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:16:27 smithi146 ceph-mon[35311]: pgmap v222: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 341 B/s wr, 0 op/s 2024-05-04T12:16:27.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:16:27 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-04T12:16:28.506 INFO:tasks.workunit.client.0.smithi079.stderr:Updating files: 90% (10989/12088) Updating files: 91% (11001/12088) Updating files: 92% (11121/12088) Updating files: 93% (11242/12088) Updating files: 94% (11363/12088) Updating files: 95% (11484/12088) Updating files: 96% (11605/12088) Updating files: 97% (11726/12088) Updating files: 98% (11847/12088) Updating files: 99% (11968/12088) Updating files: 100% (12088/12088) Updating files: 100% (12088/12088), done. 2024-05-04T12:16:29.321 INFO:tasks.workunit.client.0.smithi079.stderr:Note: switching to '09dbd6bba52d19ba471c33bc15c008e6ad158ea6'. 2024-05-04T12:16:29.322 INFO:tasks.workunit.client.0.smithi079.stderr: 2024-05-04T12:16:29.322 INFO:tasks.workunit.client.0.smithi079.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-05-04T12:16:29.322 INFO:tasks.workunit.client.0.smithi079.stderr:changes and commit them, and you can discard any commits you make in this 2024-05-04T12:16:29.322 INFO:tasks.workunit.client.0.smithi079.stderr:state without impacting any branches by switching back to a branch. 2024-05-04T12:16:29.322 INFO:tasks.workunit.client.0.smithi079.stderr: 2024-05-04T12:16:29.322 INFO:tasks.workunit.client.0.smithi079.stderr:If you want to create a new branch to retain commits you create, you may 2024-05-04T12:16:29.322 INFO:tasks.workunit.client.0.smithi079.stderr:do so (now or later) by using -c with the switch command. Example: 2024-05-04T12:16:29.322 INFO:tasks.workunit.client.0.smithi079.stderr: 2024-05-04T12:16:29.322 INFO:tasks.workunit.client.0.smithi079.stderr: git switch -c 2024-05-04T12:16:29.322 INFO:tasks.workunit.client.0.smithi079.stderr: 2024-05-04T12:16:29.322 INFO:tasks.workunit.client.0.smithi079.stderr:Or undo this operation with: 2024-05-04T12:16:29.322 INFO:tasks.workunit.client.0.smithi079.stderr: 2024-05-04T12:16:29.322 INFO:tasks.workunit.client.0.smithi079.stderr: git switch - 2024-05-04T12:16:29.323 INFO:tasks.workunit.client.0.smithi079.stderr: 2024-05-04T12:16:29.323 INFO:tasks.workunit.client.0.smithi079.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-05-04T12:16:29.323 INFO:tasks.workunit.client.0.smithi079.stderr: 2024-05-04T12:16:29.323 INFO:tasks.workunit.client.0.smithi079.stderr:HEAD is now at 09dbd6bba52 Merge branch 'wip-65150-squid' of https://github.com/sseshasa/ceph into wip-yuri4-testing-2024-04-29-0642 2024-05-04T12:16:29.327 DEBUG:teuthology.orchestra.run.smithi079:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-05-04T12:16:29.367 INFO:tasks.workunit.client.0.smithi079.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-05-04T12:16:29.370 INFO:tasks.workunit.client.0.smithi079.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-05-04T12:16:29.370 INFO:tasks.workunit.client.0.smithi079.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-05-04T12:16:29.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:16:29 smithi079 ceph-mon[27222]: pgmap v223: 65 pgs: 65 active+clean; 758 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 15 KiB/s wr, 1 op/s 2024-05-04T12:16:29.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:16:29 smithi146 ceph-mon[35311]: pgmap v223: 65 pgs: 65 active+clean; 758 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 15 KiB/s wr, 1 op/s 2024-05-04T12:16:30.014 INFO:tasks.workunit.client.0.smithi079.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-05-04T12:16:30.066 INFO:tasks.workunit.client.0.smithi079.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-05-04T12:16:30.107 INFO:tasks.workunit.client.0.smithi079.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-05-04T12:16:30.109 INFO:tasks.workunit.client.0.smithi079.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-05-04T12:16:30.109 INFO:tasks.workunit.client.0.smithi079.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-05-04T12:16:30.147 INFO:tasks.workunit.client.0.smithi079.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-05-04T12:16:30.150 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-05-04T12:16:30.150 DEBUG:teuthology.orchestra.run.smithi079:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-05-04T12:16:30.219 INFO:tasks.workunit:Running workunits matching suites/fsstress.sh on client.0... 2024-05-04T12:16:30.220 INFO:tasks.workunit:Running workunit suites/fsstress.sh... 2024-05-04T12:16:30.220 DEBUG:teuthology.orchestra.run.smithi079:workunit test suites/fsstress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=09dbd6bba52d19ba471c33bc15c008e6ad158ea6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh 2024-05-04T12:16:30.302 INFO:tasks.workunit.client.0.smithi079.stderr:+ mkdir -p fsstress 2024-05-04T12:16:30.307 INFO:tasks.workunit.client.0.smithi079.stderr:+ pushd fsstress 2024-05-04T12:16:30.307 INFO:tasks.workunit.client.0.smithi079.stdout:~/cephtest/mnt.0/client.0/tmp/fsstress ~/cephtest/mnt.0/client.0/tmp 2024-05-04T12:16:30.307 INFO:tasks.workunit.client.0.smithi079.stderr:+ wget -q -O ltp-full.tgz http://download.ceph.com/qa/ltp-full-20091231.tgz 2024-05-04T12:16:31.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:16:31 smithi079 ceph-mon[27222]: pgmap v224: 65 pgs: 65 active+clean; 758 KiB data, 172 MiB used, 536 GiB / 536 GiB avail; 15 KiB/s wr, 0 op/s 2024-05-04T12:16:31.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:16:31 smithi146 ceph-mon[35311]: pgmap v224: 65 pgs: 65 active+clean; 758 KiB data, 172 MiB used, 536 GiB / 536 GiB avail; 15 KiB/s wr, 0 op/s 2024-05-04T12:16:32.394 INFO:tasks.workunit.client.0.smithi079.stderr:+ tar xzf ltp-full.tgz 2024-05-04T12:16:33.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:16:33 smithi079 ceph-mon[27222]: pgmap v225: 65 pgs: 65 active+clean; 16 MiB data, 228 MiB used, 536 GiB / 536 GiB avail; 1.3 MiB/s wr, 115 op/s 2024-05-04T12:16:33.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:16:33 smithi146 ceph-mon[35311]: pgmap v225: 65 pgs: 65 active+clean; 16 MiB data, 228 MiB used, 536 GiB / 536 GiB avail; 1.3 MiB/s wr, 115 op/s 2024-05-04T12:16:35.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:16:35 smithi079 ceph-mon[27222]: pgmap v226: 65 pgs: 65 active+clean; 41 MiB data, 314 MiB used, 536 GiB / 536 GiB avail; 3.6 MiB/s wr, 335 op/s 2024-05-04T12:16:35.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:16:35 smithi146 ceph-mon[35311]: pgmap v226: 65 pgs: 65 active+clean; 41 MiB data, 314 MiB used, 536 GiB / 536 GiB avail; 3.6 MiB/s wr, 335 op/s 2024-05-04T12:16:37.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:16:37 smithi079 ceph-mon[27222]: pgmap v227: 65 pgs: 65 active+clean; 50 MiB data, 339 MiB used, 536 GiB / 536 GiB avail; 4.4 MiB/s wr, 413 op/s 2024-05-04T12:16:37.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:16:37 smithi146 ceph-mon[35311]: pgmap v227: 65 pgs: 65 active+clean; 50 MiB data, 339 MiB used, 536 GiB / 536 GiB avail; 4.4 MiB/s wr, 413 op/s 2024-05-04T12:16:39.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:16:39 smithi079 ceph-mon[27222]: pgmap v228: 65 pgs: 65 active+clean; 63 MiB data, 381 MiB used, 536 GiB / 536 GiB avail; 5.5 MiB/s wr, 457 op/s 2024-05-04T12:16:39.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:16:39 smithi146 ceph-mon[35311]: pgmap v228: 65 pgs: 65 active+clean; 63 MiB data, 381 MiB used, 536 GiB / 536 GiB avail; 5.5 MiB/s wr, 457 op/s 2024-05-04T12:16:41.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:16:41 smithi079 ceph-mon[27222]: pgmap v229: 65 pgs: 65 active+clean; 74 MiB data, 438 MiB used, 536 GiB / 536 GiB avail; 6.4 MiB/s wr, 515 op/s 2024-05-04T12:16:41.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:16:41 smithi146 ceph-mon[35311]: pgmap v229: 65 pgs: 65 active+clean; 74 MiB data, 438 MiB used, 536 GiB / 536 GiB avail; 6.4 MiB/s wr, 515 op/s 2024-05-04T12:16:42.428 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:16:42 smithi146 ceph-mon[35311]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-04T12:16:42.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:16:42 smithi079 ceph-mon[27222]: from='mgr.14219 172.21.15.79:0/565835706' entity='mgr.smithi079.dgcpwl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-04T12:16:43.446 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:16:43 smithi079 ceph-mon[27222]: pgmap v230: 65 pgs: 65 active+clean; 86 MiB data, 475 MiB used, 536 GiB / 536 GiB avail; 7.5 MiB/s wr, 551 op/s 2024-05-04T12:16:43.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:16:43 smithi146 ceph-mon[35311]: pgmap v230: 65 pgs: 65 active+clean; 86 MiB data, 475 MiB used, 536 GiB / 536 GiB avail; 7.5 MiB/s wr, 551 op/s 2024-05-04T12:16:45.018 INFO:tasks.workunit.client.1.smithi146.stderr:+ pushd ltp-full-20091231/testcases/kernel/fs/fsstress 2024-05-04T12:16:45.019 INFO:tasks.workunit.client.1.smithi146.stdout:~/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress ~/cephtest/mnt.1/client.1/tmp/fsstress ~/cephtest/mnt.1/client.1/tmp 2024-05-04T12:16:45.019 INFO:tasks.workunit.client.1.smithi146.stderr:+ make 2024-05-04T12:16:45.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:16:45 smithi079 ceph-mon[27222]: pgmap v231: 65 pgs: 65 active+clean; 90 MiB data, 495 MiB used, 536 GiB / 536 GiB avail; 6.5 MiB/s wr, 442 op/s 2024-05-04T12:16:45.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:16:45 smithi146 ceph-mon[35311]: pgmap v231: 65 pgs: 65 active+clean; 90 MiB data, 495 MiB used, 536 GiB / 536 GiB avail; 6.5 MiB/s wr, 442 op/s 2024-05-04T12:16:46.988 INFO:tasks.workunit.client.1.smithi146.stdout:cc -DNO_XFS -I/home/ubuntu/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress -D_LARGEFILE64_SOURCE -D_GNU_SOURCE -I../../../../include -I../../../../include -L../../../../lib fsstress.c -o fsstress 2024-05-04T12:16:47.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:16:47 smithi079 ceph-mon[27222]: pgmap v232: 65 pgs: 65 active+clean; 95 MiB data, 511 MiB used, 536 GiB / 536 GiB avail; 4.7 MiB/s wr, 223 op/s 2024-05-04T12:16:47.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:16:47 smithi146 ceph-mon[35311]: pgmap v232: 65 pgs: 65 active+clean; 95 MiB data, 511 MiB used, 536 GiB / 536 GiB avail; 4.7 MiB/s wr, 223 op/s 2024-05-04T12:16:49.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:16:49 smithi079 ceph-mon[27222]: pgmap v233: 65 pgs: 65 active+clean; 111 MiB data, 571 MiB used, 536 GiB / 536 GiB avail; 5.3 MiB/s wr, 361 op/s 2024-05-04T12:16:49.527 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:May 04 12:16:49 smithi146 ceph-mon[35311]: pgmap v233: 65 pgs: 65 active+clean; 111 MiB data, 571 MiB used, 536 GiB / 536 GiB avail; 5.3 MiB/s wr, 361 op/s 2024-05-04T12:16:51.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:16:51 smithi079 ceph-mon[27222]: pgmap v234: 65 pgs: 65 active+clean; 134 MiB data, 627 MiB used, 536 GiB / 536 GiB avail; 6.3 MiB/s wr, 535 op/s 2024-05-04T12:17:15.444 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:15.189+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:15.444 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:15.189+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:15.444 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:15.189+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:16.284 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d9b985c4-0a0e-11ef-bc97-c7b262605968 -e sha1=09dbd6bba52d19ba471c33bc15c008e6ad158ea6 -- bash -c 'ceph orch ps' 2024-05-04T12:17:16.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:17:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:17:16.112+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:17:16.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:17:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:17:16.112+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:17:16.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:17:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:17:16.112+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:17:16.446 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:16.230+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:16.446 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:16.230+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:16.446 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:16.230+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:16.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:17:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:17:16.315+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:17:16.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:17:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:17:16.315+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:17:16.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:17:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:17:16.315+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:17:16.545 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/d9b985c4-0a0e-11ef-bc97-c7b262605968/mon.smithi079/config 2024-05-04T12:17:17.341 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:17.196+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:17.341 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:17.196+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:17.341 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:17.196+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:17.341 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:17:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:17:17.159+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:17:17.341 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:17:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:17:17.159+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:17:17.341 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:17:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:17:17.159+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:17:17.695 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:17:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:17:17.341+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:17:17.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:17:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:17:17.341+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:17:17.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:17:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:17:17.341+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:17:18.304 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:17:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:17:18.144+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:17:18.304 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:17:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:17:18.144+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:17:18.304 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:17:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:17:18.144+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:17:18.304 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:17:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:17:18.304+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:17:18.305 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:17:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:17:18.304+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:17:18.305 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:17:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:17:18.304+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:17:18.305 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:18.174+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:18.305 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:18.174+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:18.305 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:18.174+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:19.395 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:19.216+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:19.396 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:19.216+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:19.396 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:19.216+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:19.396 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:17:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:17:19.116+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:17:19.396 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:17:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:17:19.116+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:17:19.396 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:17:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:17:19.116+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:17:19.397 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:17:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:17:19.325+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:17:19.397 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:17:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:17:19.325+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:17:19.397 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:17:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:17:19.325+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:17:20.446 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:20.243+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:20.446 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:20.243+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:20.446 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:20.243+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:20.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:17:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:17:20.357+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:17:20.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:17:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:17:20.357+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:17:20.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:17:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:17:20.357+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:17:20.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:17:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:17:20.155+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:17:20.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:17:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:17:20.155+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:17:20.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:17:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:17:20.155+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:17:21.336 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:21.197+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:21.336 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:21.197+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:21.336 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:21.197+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:21.336 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:17:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:17:21.196+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:17:21.336 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:17:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:17:21.196+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:17:21.336 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:17:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:17:21.196+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:17:21.695 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:17:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:17:21.336+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:17:21.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:17:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:17:21.336+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:17:21.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:17:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:17:21.336+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:17:22.695 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:17:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:17:22.315+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:17:22.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:17:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:17:22.315+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:17:22.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:17:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:17:22.315+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:17:22.696 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:22.218+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:22.696 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:22.218+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:22.696 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:22.218+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:22.696 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:17:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:17:22.243+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:17:22.696 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:17:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:17:22.243+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:17:22.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:17:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:17:22.243+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:17:23.505 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:17:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:17:23.200+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:17:23.505 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:17:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:17:23.200+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:17:23.505 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:17:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:17:23.200+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:17:23.505 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:17:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:17:23.268+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:17:23.505 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:17:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:17:23.268+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:17:23.505 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:17:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:17:23.268+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:17:23.506 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:23.218+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:23.506 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:23.218+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:23.506 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:23.218+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:24.695 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:24.231+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:24.696 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:24.231+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:24.696 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:24.231+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:24.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:17:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:17:24.228+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:17:24.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:17:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:17:24.228+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:17:24.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:17:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:17:24.228+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:17:24.696 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:17:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:17:24.249+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:17:24.696 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:17:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:17:24.249+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:17:24.696 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:17:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:17:24.249+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:17:25.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:17:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:17:25.218+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:17:25.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:17:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:17:25.218+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:17:25.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:17:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:17:25.218+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:17:25.696 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:25.274+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:25.696 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:25.274+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:25.696 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:25.274+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:25.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:17:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:17:25.281+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:17:25.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:17:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:17:25.281+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:17:25.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:17:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:17:25.281+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:17:26.445 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:17:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:17:26.246+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:17:26.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:17:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:17:26.246+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:17:26.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:17:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:17:26.246+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:17:26.446 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:26.238+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:26.446 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:26.238+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:26.446 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:26.238+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:26.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:17:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:17:26.314+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:17:26.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:17:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:17:26.314+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:17:26.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:17:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:17:26.314+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:17:27.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:17:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:17:27.270+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:17:27.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:17:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:17:27.270+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:17:27.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:17:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:17:27.270+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:17:27.696 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:27.287+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:27.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:27.287+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:27.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:27.287+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:27.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:17:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:17:27.364+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:17:27.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:17:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:17:27.364+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:17:27.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:17:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:17:27.364+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:17:28.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:17:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:17:28.224+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:17:28.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:17:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:17:28.224+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:17:28.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:17:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:17:28.224+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:17:28.696 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:28.262+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:28.696 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:28.262+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:28.696 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:28.262+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:28.696 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:17:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:17:28.384+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:17:28.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:17:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:17:28.384+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:17:28.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:17:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:17:28.384+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:17:29.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:17:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:17:29.262+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:17:29.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:17:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:17:29.262+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:17:29.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:17:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:17:29.262+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:17:29.696 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:29.278+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:29.696 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:29.278+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:29.696 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:29.278+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:29.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:17:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:17:29.335+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:17:29.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:17:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:17:29.335+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:17:29.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:17:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:17:29.335+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:17:30.695 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:17:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:17:30.304+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:17:30.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:17:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:17:30.304+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:17:30.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:17:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:17:30.304+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:17:30.696 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:30.256+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:30.696 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:30.256+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:30.696 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:30.256+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:30.696 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:17:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:17:30.353+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:17:30.696 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:17:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:17:30.353+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:17:30.696 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:17:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:17:30.353+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:17:31.377 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:17:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:17:31.327+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:17:31.377 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:17:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:17:31.327+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:17:31.377 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:17:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:17:31.327+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:17:31.377 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:31.221+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:31.377 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:31.221+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:31.377 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:31.221+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:31.696 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:17:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:17:31.377+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:17:31.696 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:17:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:17:31.377+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:17:31.696 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:17:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:17:31.377+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:17:32.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:17:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:17:32.376+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:17:32.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:17:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:17:32.376+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:17:32.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:17:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:17:32.376+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:17:32.446 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:32.195+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:32.446 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:32.195+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:32.446 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:32.195+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:32.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:17:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:17:32.394+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:17:32.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:17:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:17:32.394+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:17:32.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:17:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:17:32.394+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:17:33.446 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:33.167+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:33.446 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:33.167+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:33.446 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:33.167+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:33.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:17:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:17:33.395+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:17:33.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:17:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:17:33.395+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:17:33.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:17:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:17:33.395+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:17:33.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:17:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:17:33.386+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:17:33.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:17:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:17:33.386+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:17:33.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:17:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:17:33.386+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:17:34.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:17:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:17:34.338+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:17:34.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:17:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:17:34.338+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:17:34.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:17:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:17:34.338+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:17:34.446 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:34.174+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:34.446 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:34.174+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:34.446 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:34.174+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:34.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:17:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:17:34.346+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:17:34.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:17:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:17:34.346+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:17:34.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:17:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:17:34.346+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:17:35.446 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:35.194+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:35.446 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:35.194+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:35.446 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:35.194+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:35.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:17:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:17:35.312+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:17:35.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:17:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:17:35.312+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:17:35.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:17:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:17:35.312+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:17:35.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:17:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:17:35.309+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:17:35.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:17:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:17:35.309+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:17:35.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:17:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:17:35.309+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:17:36.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:17:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:17:36.276+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:17:36.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:17:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:17:36.276+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:17:36.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:17:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:17:36.276+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:17:36.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:17:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:17:36.317+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:17:36.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:17:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:17:36.317+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:17:36.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:17:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:17:36.317+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:17:36.446 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:36.192+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:36.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:36.192+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:36.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:36.192+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:37.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:17:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:17:37.320+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:17:37.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:17:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:17:37.320+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:17:37.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:17:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:17:37.320+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:17:37.696 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:37.215+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:37.696 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:37.215+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:37.696 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:37.215+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:37.696 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:17:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:17:37.257+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:17:37.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:17:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:17:37.257+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:17:37.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:17:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:17:37.257+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:17:38.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:17:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:17:38.287+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:17:38.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:17:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:17:38.287+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:17:38.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:17:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:17:38.287+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:17:38.696 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:38.238+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:38.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:38.238+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:38.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:38.238+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:38.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:17:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:17:38.302+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:17:38.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:17:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:17:38.302+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:17:38.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:17:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:17:38.302+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:17:39.614 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:39.218+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:39.614 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:39.218+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:39.614 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:39.218+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:39.614 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:17:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:17:39.288+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:17:39.614 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:17:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:17:39.288+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:17:39.614 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:17:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:17:39.288+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:17:39.615 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:17:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:17:39.272+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:17:39.615 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:17:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:17:39.272+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:17:39.615 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:17:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:17:39.272+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:17:40.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:17:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:17:40.311+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:17:40.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:17:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:17:40.311+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:17:40.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:17:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:17:40.311+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:17:40.696 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:40.212+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:40.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:40.212+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:40.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:40.212+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:40.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:17:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:17:40.246+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:17:40.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:17:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:17:40.246+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:17:40.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:17:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:17:40.246+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:17:41.446 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:41.202+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:41.446 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:41.202+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:41.446 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:41.202+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:41.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:17:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:17:41.266+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:17:41.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:17:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:17:41.266+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:17:41.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:17:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:17:41.266+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:17:41.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:17:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:17:41.227+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:17:41.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:17:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:17:41.227+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:17:41.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:17:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:17:41.227+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:17:42.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:17:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:17:42.218+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:17:42.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:17:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:17:42.218+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:17:42.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:17:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:17:42.218+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:17:42.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:17:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:17:42.259+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:17:42.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:17:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:17:42.259+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:17:42.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:17:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:17:42.259+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:17:42.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:42.156+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:42.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:42.156+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:42.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:42.156+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:43.507 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:43.198+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:43.507 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:43.198+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:43.507 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:43.198+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:43.507 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:17:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:17:43.250+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:17:43.507 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:17:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:17:43.250+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:17:43.507 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:17:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:17:43.250+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:17:43.508 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:17:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:17:43.248+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:17:43.508 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:17:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:17:43.248+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:17:43.508 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:17:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:17:43.248+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:17:44.661 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:17:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:17:44.216+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:17:44.661 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:17:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:17:44.216+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:17:44.661 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:17:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:17:44.216+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:17:44.661 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:44.246+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:44.661 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:44.246+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:44.662 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:44.246+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:44.662 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:17:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:17:44.239+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:17:44.662 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:17:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:17:44.239+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:17:44.662 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:17:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:17:44.239+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:17:45.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:17:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:17:45.198+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:17:45.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:17:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:17:45.198+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:17:45.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:17:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:17:45.198+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:17:45.696 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:45.255+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:45.696 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:45.255+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:45.696 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:45.255+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:45.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:17:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:17:45.277+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:17:45.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:17:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:17:45.277+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:17:45.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:17:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:17:45.277+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:17:46.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:17:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:17:46.153+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:17:46.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:17:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:17:46.153+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:17:46.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:17:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:17:46.153+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:17:46.446 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:46.231+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:46.446 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:46.231+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:46.446 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:46.231+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:46.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:17:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:17:46.261+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:17:46.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:17:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:17:46.261+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:17:46.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:17:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:17:46.261+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:17:47.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:17:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:17:47.113+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:17:47.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:17:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:17:47.113+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:17:47.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:17:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:17:47.113+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:17:47.446 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:47.270+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:47.446 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:47.270+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:47.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:47.270+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:47.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:17:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:17:47.238+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:17:47.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:17:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:17:47.238+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:17:47.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:17:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:17:47.238+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:17:48.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:17:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:17:48.078+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:17:48.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:17:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:17:48.078+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:17:48.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:17:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:17:48.078+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:17:48.446 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:48.276+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:48.446 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:48.276+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:48.446 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:48.276+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:48.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:17:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:17:48.203+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:17:48.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:17:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:17:48.203+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:17:48.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:17:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:17:48.203+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:17:49.324 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:17:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:17:49.216+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:17:49.324 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:17:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:17:49.216+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:17:49.324 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:17:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:17:49.216+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:17:49.324 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:17:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:17:49.055+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:17:49.324 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:17:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:17:49.055+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:17:49.324 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:17:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:17:49.055+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:17:49.695 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:49.325+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:49.696 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:49.325+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:49.696 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:49.325+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:50.346 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:17:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:17:50.090+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:17:50.346 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:17:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:17:50.090+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:17:50.346 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:17:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:17:50.090+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:17:50.346 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:17:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:17:50.187+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:17:50.346 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:17:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:17:50.187+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:17:50.346 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:17:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:17:50.187+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:17:50.695 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:50.345+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:50.696 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:50.346+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:50.696 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:50.346+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:51.382 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:17:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:17:51.059+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:17:51.382 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:17:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:17:51.059+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:17:51.382 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:17:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:17:51.059+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:17:51.382 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:17:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:17:51.182+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:17:51.382 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:17:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:17:51.182+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:17:51.383 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:17:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:17:51.182+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:17:51.695 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:51.382+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:51.696 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:51.382+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:51.696 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:51.382+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:52.404 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:17:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:17:52.060+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:17:52.404 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:17:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:17:52.060+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:17:52.404 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:17:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:17:52.060+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:17:52.405 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:17:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:17:52.143+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:17:52.405 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:17:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:17:52.143+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:17:52.405 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:17:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:17:52.143+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:17:52.695 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:52.404+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:52.695 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:52.404+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:52.696 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:52.404+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:53.411 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:17:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:17:53.178+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:17:53.411 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:17:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:17:53.178+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:17:53.411 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:17:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:17:53.178+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:17:53.411 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:17:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:17:53.034+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:17:53.411 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:17:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:17:53.034+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:17:53.411 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:17:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:17:53.034+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:17:53.696 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:53.411+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:53.696 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:53.411+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:53.696 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:53.411+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:54.408 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:17:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:17:54.173+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:17:54.408 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:17:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:17:54.173+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:17:54.408 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:17:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:17:54.173+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:17:54.408 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:17:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:17:54.055+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:17:54.409 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:17:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:17:54.055+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:17:54.409 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:17:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:17:54.055+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:17:54.696 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:54.408+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:54.696 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:54.408+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:54.696 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:54.408+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:55.387 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:17:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:17:55.027+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:17:55.387 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:17:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:17:55.027+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:17:55.387 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:17:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:17:55.027+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:17:55.387 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:17:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:17:55.155+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:17:55.387 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:17:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:17:55.155+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:17:55.387 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:17:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:17:55.155+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:17:55.695 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:55.387+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:55.696 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:55.387+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:55.696 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:55.387+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:56.433 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:17:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:17:56.143+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:17:56.433 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:17:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:17:56.143+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:17:56.433 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:17:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:17:56.143+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:17:56.434 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:17:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:17:56.043+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:17:56.434 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:17:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:17:56.043+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:17:56.434 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:17:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:17:56.043+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:17:56.695 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:56.433+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:56.696 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:56.433+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:56.696 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:56.433+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:57.441 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:17:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:17:57.006+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:17:57.441 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:17:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:17:57.006+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:17:57.441 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:17:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:17:57.006+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:17:57.441 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:17:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:17:57.104+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:17:57.441 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:17:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:17:57.104+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:17:57.441 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:17:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:17:57.104+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:17:57.695 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:57.441+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:57.696 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:57.441+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:57.696 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:57.441+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:58.442 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:17:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:17:58.055+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:17:58.443 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:17:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:17:58.055+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:17:58.443 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:17:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:17:58.055+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:17:58.443 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:17:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:17:58.114+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:17:58.443 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:17:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:17:58.114+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:17:58.443 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:17:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:17:58.114+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:17:58.695 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:58.442+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:58.696 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:58.442+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:58.696 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:58.442+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:59.427 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:17:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:17:59.078+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:17:59.427 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:17:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:17:59.078+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:17:59.428 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:17:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:17:59.078+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:17:59.428 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:17:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:17:59.064+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:17:59.428 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:17:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:17:59.064+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:17:59.428 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:17:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:17:59.064+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:17:59.695 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:59.427+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:59.696 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:59.427+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:17:59.696 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:17:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:17:59.427+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:00.438 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:00.076+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:00.438 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:00.076+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:00.438 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:00.076+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:00.438 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:00.122+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:00.438 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:00.122+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:00.438 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:00.122+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:00.696 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:00.437+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:00.696 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:00.437+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:00.696 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:00.437+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:01.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:01.057+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:01.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:01.057+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:01.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:01.057+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:01.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:01.119+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:01.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:01.119+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:01.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:01.119+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:01.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:01.451+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:01.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:01.451+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:01.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:01.451+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:02.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:02.065+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:02.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:02.065+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:02.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:02.065+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:02.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:02.160+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:02.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:02.160+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:02.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:02.160+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:02.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:02.472+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:02.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:02.472+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:02.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:02.472+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:03.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:03.142+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:03.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:03.142+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:03.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:03.142+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:03.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:03.062+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:03.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:03.062+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:03.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:03.062+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:03.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:03.471+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:03.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:03.471+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:03.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:03.471+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:04.438 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:04.058+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:04.438 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:04.058+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:04.439 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:04.058+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:04.439 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:04.115+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:04.439 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:04.115+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:04.439 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:04.115+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:04.695 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:04.438+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:04.696 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:04.438+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:04.696 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:04.438+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:05.196 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:05.025+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:05.196 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:05.025+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:05.196 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:05.025+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:05.196 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:05.086+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:05.196 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:05.086+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:05.196 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:05.086+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:05.695 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:05.423+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:05.696 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:05.423+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:05.696 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:05.423+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:06.377 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:06.116+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:06.378 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:06.116+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:06.378 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:06.116+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:06.378 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:06.063+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:06.378 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:06.063+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:06.378 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:06.063+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:06.696 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:06.377+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:06.696 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:06.377+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:06.696 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:06.377+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:07.404 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:07.112+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:07.404 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:07.112+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:07.404 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:07.112+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:07.404 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:07.143+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:07.404 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:07.143+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:07.404 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:07.143+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:07.696 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:07.403+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:07.696 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:07.403+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:07.696 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:07.403+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:08.443 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:08.175+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:08.443 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:08.175+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:08.443 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:08.175+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:08.443 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:08.154+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:08.444 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:08.154+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:08.444 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:08.154+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:08.444 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:08.383+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:08.444 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:08.383+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:08.444 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:08.383+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:09.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:09.180+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:09.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:09.180+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:09.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:09.180+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:09.446 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:09.374+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:09.446 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:09.374+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:09.446 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:09.374+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:09.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:09.139+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:09.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:09.139+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:09.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:09.139+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:10.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:10.197+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:10.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:10.197+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:10.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:10.197+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:10.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:10.177+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:10.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:10.177+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:10.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:10.177+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:10.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:10.419+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:10.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:10.419+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:10.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:10.419+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:11.400 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:11.138+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:11.400 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:11.138+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:11.400 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:11.138+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:11.400 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:11.201+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:11.401 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:11.201+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:11.401 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:11.201+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:11.696 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:11.400+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:11.696 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:11.400+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:11.696 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:11.400+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:12.384 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:12.100+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:12.384 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:12.100+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:12.384 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:12.100+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:12.384 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:12.243+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:12.384 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:12.243+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:12.385 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:12.243+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:12.695 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:12.384+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:12.696 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:12.384+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:12.696 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:12.384+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:13.401 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:13.255+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:13.401 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:13.255+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:13.402 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:13.255+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:13.402 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:13.134+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:13.402 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:13.134+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:13.402 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:13.134+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:13.696 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:13.401+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:13.696 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:13.401+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:13.696 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:13.401+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:14.442 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:14.294+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:14.442 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:14.294+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:14.442 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:14.294+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:14.442 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:14.139+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:14.442 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:14.139+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:14.442 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:14.139+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:14.695 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:14.441+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:14.696 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:14.441+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:14.696 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:14.441+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:15.261 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:15.117+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:15.261 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:15.117+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:15.261 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:15.117+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:15.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:15.261+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:15.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:15.261+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:15.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:15.261+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:15.696 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:15.447+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:15.696 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:15.447+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:15.696 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:15.447+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:16.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:16.276+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:16.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:16.276+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:16.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:16.276+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:16.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:16.070+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:16.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:16.070+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:16.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:16.070+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:16.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:16.452+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:16.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:16.452+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:16.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:16.452+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:17.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:17.232+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:17.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:17.232+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:17.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:17.232+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:17.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:17.045+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:17.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:17.045+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:17.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:17.045+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:17.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:17.462+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:17.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:17.462+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:17.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:17.462+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:18.425 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:18.242+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:18.425 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:18.242+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:18.426 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:18.242+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:18.426 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:18.065+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:18.426 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:18.065+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:18.426 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:18.065+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:18.696 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:18.425+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:18.696 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:18.425+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:18.696 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:18.425+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:19.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:19.290+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:19.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:19.290+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:19.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:19.290+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:19.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:19.074+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:19.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:19.074+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:19.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:19.074+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:19.831 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:19.466+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:19.831 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:19.466+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:19.831 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:19.466+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:20.196 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:20.024+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:20.196 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:20.024+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:20.196 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:20.024+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:20.494 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:20.241+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:20.494 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:20.241+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:20.494 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:20.241+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:20.945 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:20.494+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:20.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:20.494+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:20.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:20.494+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:21.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:21.202+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:21.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:21.202+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:21.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:21.202+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:21.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:21.002+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:21.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:21.002+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:21.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:21.002+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:21.945 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:21.460+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:21.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:21.460+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:21.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:21.460+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:22.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:22.026+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:22.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:22.026+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:22.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:22.026+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:22.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:22.201+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:22.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:22.201+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:22.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:22.201+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:22.945 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:22.490+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:22.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:22.490+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:22.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:22.490+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:23.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:23.226+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:23.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:23.226+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:23.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:23.226+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:23.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:23.031+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:23.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:23.031+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:23.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:23.031+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:23.802 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:23.473+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:23.802 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:23.473+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:23.802 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:23.473+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:24.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:24.056+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:24.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:24.056+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:24.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:24.056+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:24.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:24.180+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:24.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:24.180+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:24.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:24.180+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:24.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:24.513+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:24.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:24.513+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:24.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:24.513+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:25.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:25.040+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:25.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:25.040+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:25.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:25.040+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:25.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:25.140+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:25.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:25.140+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:25.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:25.140+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:25.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:25.491+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:25.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:25.491+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:25.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:25.491+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:26.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:26.012+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:26.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:26.012+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:26.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:26.012+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:26.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:26.127+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:26.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:26.127+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:26.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:26.127+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:26.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:26.533+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:26.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:26.533+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:26.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:26.533+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:27.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:27.153+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:27.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:27.153+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:27.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:27.153+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:27.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:26.978+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:27.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:26.978+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:27.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:26.978+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:27.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:27.531+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:27.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:27.531+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:27.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:27.531+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:28.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:28.132+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:28.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:28.132+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:28.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:28.132+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:28.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:27.978+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:28.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:27.978+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:28.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:27.978+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:28.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:28.483+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:28.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:28.483+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:28.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:28.483+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:29.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:29.106+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:29.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:29.106+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:29.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:29.106+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:29.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:28.976+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:29.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:28.976+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:29.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:28.976+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:29.831 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:29.525+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:29.831 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:29.525+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:29.831 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:29.525+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:30.104 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:29.943+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:30.104 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:29.943+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:30.104 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:29.943+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:30.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:30.129+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:30.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:30.129+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:30.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:30.129+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:30.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:30.476+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:30.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:30.476+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:30.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:30.476+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:31.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:30.975+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:31.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:30.975+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:31.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:30.975+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:31.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:31.086+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:31.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:31.086+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:31.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:31.086+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:31.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:31.489+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:31.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:31.489+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:31.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:31.489+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:32.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:32.004+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:32.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:32.004+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:32.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:32.004+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:32.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:32.095+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:32.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:32.095+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:32.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:32.095+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:32.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:32.504+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:32.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:32.504+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:32.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:32.504+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:33.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:32.955+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:33.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:32.955+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:33.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:32.955+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:33.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:33.048+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:33.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:33.048+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:33.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:33.048+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:33.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:33.518+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:33.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:33.518+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:33.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:33.518+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:34.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:34.057+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:34.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:34.057+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:34.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:34.057+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:34.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:33.971+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:34.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:33.971+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:34.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:33.971+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:34.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:34.556+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:34.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:34.556+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:34.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:34.556+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:35.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:34.964+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:35.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:34.964+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:35.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:34.964+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:35.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:35.019+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:35.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:35.019+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:35.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:35.019+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:35.922 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:35.545+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:35.922 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:35.545+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:35.922 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:35.545+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:36.196 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:35.921+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:36.196 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:35.921+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:36.196 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:35.921+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:36.196 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:36.045+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:36.196 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:36.045+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:36.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:36.046+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:36.920 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:36.594+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:36.920 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:36.594+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:36.920 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:36.594+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:37.196 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:36.920+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:37.196 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:36.920+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:37.196 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:36.920+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:37.196 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:37.032+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:37.196 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:37.032+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:37.196 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:37.032+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:37.925 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:37.603+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:37.925 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:37.603+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:37.925 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:37.603+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:38.196 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:37.925+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:38.196 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:37.925+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:38.196 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:37.925+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:38.196 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:38.021+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:38.196 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:38.021+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:38.196 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:38.021+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:38.902 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:38.555+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:38.903 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:38.555+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:38.903 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:38.555+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:39.196 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:39.013+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:39.196 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:39.013+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:39.196 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:39.013+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:39.196 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:38.902+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:39.196 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:38.902+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:39.196 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:38.902+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:39.831 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:39.572+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:39.831 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:39.572+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:39.832 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:39.572+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:40.196 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:39.952+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:40.196 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:39.952+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:40.196 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:39.952+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:40.196 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:40.054+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:40.196 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:40.054+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:40.196 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:40.055+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:40.937 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:40.609+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:40.937 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:40.609+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:40.937 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:40.609+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:41.196 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:40.937+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:41.196 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:40.937+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:41.196 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:40.937+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:41.196 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:41.044+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:41.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:41.044+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:41.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:41.044+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:41.913 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:41.561+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:41.913 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:41.561+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:41.913 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:41.561+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:42.196 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:42.005+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:42.196 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:42.005+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:42.196 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:42.005+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:42.196 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:41.913+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:42.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:41.913+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:42.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:41.913+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:42.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:42.594+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:42.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:42.594+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:42.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:42.594+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:43.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:42.960+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:43.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:42.960+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:43.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:42.960+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:43.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:43.011+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:43.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:43.011+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:43.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:43.011+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:43.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:43.623+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:43.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:43.623+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:43.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:43.623+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:44.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:43.981+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:44.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:43.981+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:44.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:43.981+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:44.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:43.992+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:44.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:43.992+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:44.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:43.992+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:44.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:44.621+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:44.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:44.621+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:44.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:44.621+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:45.259 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:44.972+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:45.259 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:44.972+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:45.259 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:44.972+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:45.260 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:44.950+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:45.260 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:44.950+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:45.260 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:44.950+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:45.902 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:45.575+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:45.902 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:45.575+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:45.902 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:45.575+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:46.196 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:45.902+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:46.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:45.902+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:46.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:45.902+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:46.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:45.936+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:46.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:45.936+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:46.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:45.936+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:46.916 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:46.610+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:46.916 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:46.610+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:46.916 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:46.610+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:47.196 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:46.954+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:47.196 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:46.954+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:47.196 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:46.954+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:47.196 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:46.916+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:47.196 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:46.916+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:47.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:46.916+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:47.934 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:47.634+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:47.934 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:47.634+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:47.934 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:47.634+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:48.196 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:47.963+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:48.196 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:47.963+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:48.196 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:47.963+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:48.196 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:47.934+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:48.196 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:47.934+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:48.196 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:47.934+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:48.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:48.596+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:48.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:48.596+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:48.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:48.596+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:49.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:48.946+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:49.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:48.946+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:49.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:48.946+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:49.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:48.983+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:49.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:48.983+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:49.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:48.983+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:49.831 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:49.562+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:49.831 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:49.562+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:49.831 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:49.562+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:50.196 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:49.990+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:50.196 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:49.990+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:50.196 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:49.990+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:50.196 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:50.008+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:50.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:50.008+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:50.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:50.008+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:50.696 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:50.519+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:50.696 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:50.519+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:50.696 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:50.519+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:51.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:50.988+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:51.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:50.988+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:51.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:50.988+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:51.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:51.010+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:51.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:51.010+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:51.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:51.010+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:51.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:51.481+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:51.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:51.481+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:51.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:51.481+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:52.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:51.996+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:52.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:51.996+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:52.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:51.996+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:52.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:52.002+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:52.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:52.002+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:52.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:52.003+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:52.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:52.526+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:52.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:52.526+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:52.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:52.526+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:53.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:52.979+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:53.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:52.979+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:53.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:52.979+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:53.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:52.977+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:53.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:52.977+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:53.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:52.977+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:53.938 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:53.572+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:53.938 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:53.572+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:53.938 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:53.572+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:54.196 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:53.991+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:54.196 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:53.991+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:54.196 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:53.991+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:54.196 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:53.937+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:54.196 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:53.937+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:54.196 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:53.937+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:54.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:54.590+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:54.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:54.590+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:54.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:54.590+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:55.368 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:54.974+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:55.368 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:54.974+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:55.368 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:54.974+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:55.368 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:54.982+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:55.368 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:54.982+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:55.368 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:54.982+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:55.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:55.632+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:55.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:55.632+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:55.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:55.632+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:56.446 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:18:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:18:56.084+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 4 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:18:56.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:55.982+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:56.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:55.982+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:56.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:55.982+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:56.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:56.027+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:56.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:56.027+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:56.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:56.027+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:56.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:56.591+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:56.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:56.591+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:56.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:56.591+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:57.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:57.026+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:57.463 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:57.026+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:57.463 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:57.026+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:57.463 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:57.071+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:57.464 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:57.071+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:57.464 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:57.071+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:57.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:57.612+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:57.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:57.612+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:57.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:57.612+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:58.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:57.991+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:58.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:57.991+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:58.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:57.991+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:58.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:58.099+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:58.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:58.099+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:58.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:58.099+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:58.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:58.579+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:58.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:58.579+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:58.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:58.579+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:59.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:59.030+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:59.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:59.030+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:59.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:18:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:18:59.030+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:18:59.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:59.137+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:59.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:59.137+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:59.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:18:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:18:59.137+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:18:59.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:59.620+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:59.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:59.620+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:18:59.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:18:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:18:59.620+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:00.418 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:00.127+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:00.418 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:00.127+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:00.418 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:00.127+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:00.418 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:00.016+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:00.418 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:00.016+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:00.418 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:00.016+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:00.696 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:00.611+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:00.696 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:00.611+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:00.696 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:00.611+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:01.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:01.078+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:01.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:01.078+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:01.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:01.078+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:01.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:01.057+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:01.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:01.058+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:01.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:01.058+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:01.447 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:19:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:19:01.084+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 16 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:19:01.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:01.632+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:01.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:01.632+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:01.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:01.632+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:02.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:02.069+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:02.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:02.069+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:02.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:02.069+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:02.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:02.041+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:02.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:02.041+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:02.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:02.041+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:02.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:02.604+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:02.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:02.604+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:02.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:02.604+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:03.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:03.040+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:03.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:03.040+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:03.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:03.040+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:03.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:03.095+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:03.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:03.095+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:03.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:03.095+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:03.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:03.601+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:03.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:03.601+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:03.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:03.601+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:04.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:04.079+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:04.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:04.079+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:04.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:04.079+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:04.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:04.127+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:04.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:04.127+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:04.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:04.127+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:04.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:04.618+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:04.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:04.618+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:04.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:04.618+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:05.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:05.166+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:05.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:05.166+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:05.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:05.166+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:05.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:05.051+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:05.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:05.051+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:05.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:05.051+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:05.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:05.607+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:05.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:05.607+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:05.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:05.607+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:06.446 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:19:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:19:06.085+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 27 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:19:06.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:06.055+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:06.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:06.055+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:06.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:06.055+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:06.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:06.173+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:06.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:06.173+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:06.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:06.173+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:06.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:06.651+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:06.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:06.651+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:06.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:06.651+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:07.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:07.206+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:07.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:07.206+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:07.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:07.206+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:07.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:07.025+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:07.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:07.025+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:07.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:07.025+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:07.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:07.617+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:07.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:07.617+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:07.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:07.617+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:08.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:08.195+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:08.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:08.195+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:08.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:08.195+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:08.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:07.981+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:08.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:07.981+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:08.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:07.981+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:08.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:08.600+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:08.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:08.600+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:08.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:08.600+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:09.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:08.979+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:09.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:08.979+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:09.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:08.979+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:09.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:09.211+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:09.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:09.211+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:09.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:09.211+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:09.832 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:09.564+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:09.832 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:09.564+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:09.832 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:09.564+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:10.196 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:09.972+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:10.196 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:09.972+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:10.196 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:09.972+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:10.516 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:10.217+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:10.517 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:10.217+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:10.517 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:10.217+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:10.932 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:10.541+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:10.932 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:10.541+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:10.933 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:10.541+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:11.196 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:19:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:19:11.085+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 36 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:19:11.196 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:10.932+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:11.196 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:10.932+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:11.196 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:10.932+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:11.561 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:11.224+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:11.562 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:11.225+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:11.562 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:11.225+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:11.945 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:11.561+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:11.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:11.561+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:11.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:11.561+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:12.196 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:11.945+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:12.196 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:11.945+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:12.196 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:11.945+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:12.551 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:12.226+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:12.551 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:12.226+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:12.551 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:12.226+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:12.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:12.551+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:12.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:12.551+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:12.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:12.551+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:13.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:12.968+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:13.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:12.968+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:13.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:12.968+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:13.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:13.194+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:13.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:13.194+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:13.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:13.194+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:13.928 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:13.574+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:13.928 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:13.574+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:13.928 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:13.574+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:14.196 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:13.927+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:14.196 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:13.927+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:14.196 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:13.927+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:14.570 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:14.202+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:14.570 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:14.202+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:14.570 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:14.202+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:14.916 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:14.570+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:14.917 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:14.570+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:14.917 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:14.570+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:15.180 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:14.917+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:15.180 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:14.917+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:15.180 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:14.917+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:15.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:15.181+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:15.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:15.181+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:15.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:15.181+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:15.899 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:15.606+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:15.899 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:15.606+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:15.899 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:15.606+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:16.185 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:19:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:19:16.087+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 58 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:19:16.185 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:15.899+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:16.185 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:15.899+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:16.185 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:15.899+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:16.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:16.185+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:16.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:16.185+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:16.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:16.185+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:16.894 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:16.585+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:16.894 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:16.585+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:16.894 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:16.585+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:17.183 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:16.895+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:17.183 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:16.895+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:17.183 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:16.895+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:17.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:17.184+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:17.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:17.184+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:17.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:17.184+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:17.928 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:17.635+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:17.928 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:17.635+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:17.928 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:17.635+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:18.188 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:17.928+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:18.188 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:17.928+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:18.188 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:17.928+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:18.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:18.188+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:18.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:18.188+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:18.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:18.188+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:18.943 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:18.682+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:18.943 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:18.682+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:18.943 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:18.682+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:19.196 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:18.942+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:19.196 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:18.942+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:19.196 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:18.942+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:19.670 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:19.222+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:19.670 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:19.222+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:19.670 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:19.222+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:19.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:19.670+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:19.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:19.670+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:19.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:19.670+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:19.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:19.908+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:19.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:19.908+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:19.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:19.908+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:20.620 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:20.259+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:20.620 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:20.259+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:20.621 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:20.260+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:20.914 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:20.689+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:20.914 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:20.689+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:20.915 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:20.689+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:21.196 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:19:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:19:21.086+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 87 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:19:21.196 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:20.914+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:21.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:20.914+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:21.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:20.914+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:21.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:21.258+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:21.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:21.258+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:21.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:21.258+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:22.196 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:21.940+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:22.196 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:21.940+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:22.196 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:21.940+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:22.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:21.719+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:22.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:21.719+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:22.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:21.719+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:22.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:22.281+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:22.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:22.281+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:22.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:22.281+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:23.196 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:22.744+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:23.196 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:22.744+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:23.196 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:22.744+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:23.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:22.906+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:23.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:22.907+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:23.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:22.907+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:23.515 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:23.248+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:23.515 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:23.248+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:23.515 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:23.248+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:23.944 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:23.729+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:23.944 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:23.729+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:23.944 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:23.729+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:24.196 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:23.944+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:24.196 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:23.944+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:24.196 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:23.944+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:24.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:24.265+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:24.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:24.265+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:24.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:24.265+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:25.196 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:24.739+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:25.196 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:24.739+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:25.196 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:24.739+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:25.196 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:24.929+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:25.196 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:24.929+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:25.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:24.929+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:25.670 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:25.261+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:25.671 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:25.261+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:25.671 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:25.261+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:25.925 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:25.776+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:25.925 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:25.776+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:25.925 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:25.776+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:26.196 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:19:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:19:26.086+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 118 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:19:26.196 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:25.925+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:26.196 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:25.925+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:26.196 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:25.925+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:26.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:26.280+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:26.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:26.280+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:26.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:26.280+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:27.196 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:26.802+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:27.196 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:26.802+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:27.196 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:26.802+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:27.196 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:26.957+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:27.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:26.957+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:27.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:26.957+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:27.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:27.314+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:27.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:27.314+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:27.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:27.314+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:28.196 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:27.786+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:28.196 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:27.786+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:28.196 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:27.786+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:28.196 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:27.909+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:28.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:27.909+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:28.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:27.909+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:28.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:28.356+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:28.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:28.356+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:28.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:28.356+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:29.196 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:28.802+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:29.196 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:28.802+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:29.196 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:28.802+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:29.196 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:28.905+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:29.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:28.905+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:29.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:28.905+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:29.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:29.368+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:29.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:29.368+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:29.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:29.368+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:30.196 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:29.823+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:30.196 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:29.824+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:30.196 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:29.824+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:30.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:29.867+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:30.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:29.867+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:30.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:29.867+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:30.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:30.351+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:30.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:30.351+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:30.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:30.351+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:31.087 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:30.802+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:31.088 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:30.802+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:31.088 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:30.802+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:31.088 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:30.909+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:31.088 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:30.909+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:31.088 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:30.909+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:31.446 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:19:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:19:31.087+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 145 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:19:31.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:31.334+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:31.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:31.334+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:31.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:31.334+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:32.196 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:31.866+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:32.196 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:31.866+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:32.196 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:31.866+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:32.196 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:31.797+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:32.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:31.797+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:32.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:31.797+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:32.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:32.303+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:32.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:32.303+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:32.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:32.303+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:33.196 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:32.840+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:33.196 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:32.840+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:33.196 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:32.840+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:33.196 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:32.827+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:33.196 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:32.827+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:33.196 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:32.827+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:33.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:33.334+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:33.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:33.334+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:33.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:33.334+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:34.196 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:33.883+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:34.196 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:33.883+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:34.196 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:33.883+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:34.196 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:33.813+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:34.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:33.813+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:34.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:33.813+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:34.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:34.312+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:34.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:34.312+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:34.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:34.312+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:35.196 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:34.873+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:35.196 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:34.873+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:35.196 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:34.873+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:35.196 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:34.852+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:35.196 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:34.852+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:35.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:34.852+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:35.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:35.326+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:35.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:35.326+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:35.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:35.326+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:36.088 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:35.867+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:36.088 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:35.867+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:36.089 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:35.867+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:36.089 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:35.857+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:36.089 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:35.857+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:36.089 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:35.857+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:36.339 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:19:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:19:36.087+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 175 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:19:36.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:36.338+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:36.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:36.339+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:36.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:36.339+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:37.196 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:36.916+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:37.196 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:36.916+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:37.196 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:36.916+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:37.196 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:36.860+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:37.196 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:36.860+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:37.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:36.860+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:37.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:37.373+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:37.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:37.373+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:37.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:37.373+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:38.196 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:37.962+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:38.196 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:37.962+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:38.196 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:37.962+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:38.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:37.854+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:38.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:37.854+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:38.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:37.854+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:38.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:38.395+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:38.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:38.395+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:38.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:38.395+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:39.196 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:38.996+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:39.196 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:38.996+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:39.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:38.996+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:39.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:38.823+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:39.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:38.823+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:39.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:38.823+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:39.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:39.377+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:39.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:39.377+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:39.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:39.377+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:40.196 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:40.042+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:40.196 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:40.042+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:40.196 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:40.042+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:40.196 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:39.821+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:40.196 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:39.821+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:40.196 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:39.821+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:40.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:40.357+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:40.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:40.357+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:40.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:40.357+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:41.069 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:40.801+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:41.070 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:40.801+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:41.070 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:40.801+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:41.393 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:19:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:19:41.088+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 202 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:19:41.394 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:41.069+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:41.394 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:41.069+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:41.394 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:41.069+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:41.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:41.393+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:41.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:41.393+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:41.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:41.393+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:42.097 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:41.825+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:42.097 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:41.825+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:42.097 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:41.825+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:42.401 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:42.096+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:42.401 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:42.096+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:42.401 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:42.096+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:42.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:42.401+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:42.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:42.401+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:42.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:42.401+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:43.196 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:43.104+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:43.196 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:43.104+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:43.196 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:43.104+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:43.196 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:42.874+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:43.196 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:42.874+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:43.196 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:42.874+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:43.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:43.393+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:43.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:43.393+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:43.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:43.393+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:44.154 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:43.898+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:44.154 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:43.898+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:44.154 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:43.898+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:44.441 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:44.154+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:44.442 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:44.154+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:44.442 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:44.154+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:44.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:44.441+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:44.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:44.441+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:44.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:44.441+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:45.169 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:44.860+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:45.169 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:44.860+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:45.169 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:44.860+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:45.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:45.402+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:45.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:45.402+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:45.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:45.402+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:45.446 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:45.169+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:45.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:45.169+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:45.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:45.169+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:46.153 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:19:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:19:46.088+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 232 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:19:46.153 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:45.899+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:46.153 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:45.899+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:46.153 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:45.899+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:46.444 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:46.152+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:46.444 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:46.152+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:46.444 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:46.152+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:46.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:46.444+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:46.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:46.444+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:46.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:46.444+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:47.196 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:47.148+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:47.196 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:47.148+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:47.196 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:47.148+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:47.196 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:46.941+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:47.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:46.941+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:47.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:46.941+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:47.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:47.433+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:47.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:47.433+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:47.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:47.433+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:48.196 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:48.158+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:48.196 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:48.158+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:48.196 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:48.158+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:48.196 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:47.944+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:48.196 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:47.944+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:48.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:47.944+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:48.946 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:48.446+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:48.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:48.446+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:48.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:48.446+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:49.413 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:48.984+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:49.413 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:48.984+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:49.413 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:48.984+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:49.413 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:49.207+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:49.413 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:49.207+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:49.413 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:49.207+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:49.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:49.412+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:49.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:49.412+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:49.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:49.412+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:50.195 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:49.961+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:50.196 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:49.961+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:50.196 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:49.961+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:50.446 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:50.195+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:50.446 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:50.195+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:50.446 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:50.195+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:50.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:50.380+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:50.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:50.380+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:50.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:50.380+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:51.196 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:19:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:19:51.089+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 259 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:19:51.196 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:51.005+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:51.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:51.005+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:51.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:51.005+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:51.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:51.350+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:51.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:51.350+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:51.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:51.350+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:51.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:51.213+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:51.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:51.213+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:51.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:51.213+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:52.330 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:52.206+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:52.331 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:52.206+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:52.331 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:52.206+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:52.331 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:52.208+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:19:52.331 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:52.034+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:52.331 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:52.034+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:52.331 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:52.034+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:52.331 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:52.034+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.24317.0:1713 3.7 3:e53702a6:::1000000250f.00000000:head [write 0~2475 in=2475b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:19:52.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:52.330+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:52.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:52.330+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:52.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:52.330+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:52.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:52.331+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:19:53.331 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:53.227+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:53.331 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:53.227+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:53.331 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:53.227+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:53.332 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:53.227+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:19:53.332 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:53.011+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:53.332 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:53.011+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:53.332 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:53.011+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:53.332 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:53.013+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:19:53.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:53.330+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:53.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:53.330+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:53.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:53.330+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:53.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:53.331+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:19:54.338 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:54.253+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:54.338 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:54.253+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:54.338 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:54.253+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:54.338 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:54.253+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:19:54.338 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:54.034+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:54.338 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:54.034+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:54.338 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:54.034+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:54.339 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:54.035+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:19:54.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:54.337+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:54.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:54.337+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:54.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:54.337+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:54.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:54.339+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:19:55.370 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:55.225+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:55.370 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:55.225+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:55.370 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:55.225+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:55.370 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:55.226+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:19:55.370 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:55.073+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:55.371 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:55.073+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:55.371 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:55.073+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:55.371 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:55.074+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:19:55.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:55.369+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:55.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:55.369+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:55.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:55.369+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:55.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:55.370+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:19:56.363 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:19:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:19:56.089+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 291 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:19:56.363 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:56.108+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:56.363 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:56.108+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:56.363 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:56.108+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:56.363 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:56.109+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:19:56.363 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:56.184+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:56.363 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:56.184+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:56.364 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:56.184+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:56.364 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:56.185+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:19:56.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:56.362+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:56.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:56.362+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:56.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:56.362+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:56.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:56.364+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:19:57.411 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:57.083+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:57.411 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:57.083+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:57.411 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:57.083+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:57.411 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:57.083+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:19:57.411 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:57.231+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:57.412 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:57.231+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:57.412 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:57.231+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:57.412 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:57.232+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:19:57.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:57.410+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:57.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:57.410+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:57.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:57.410+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:57.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:57.411+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:19:58.382 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:58.254+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:58.382 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:58.254+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:58.383 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:58.254+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:58.383 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:58.254+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:19:58.383 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:58.110+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:58.383 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:58.110+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:58.383 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:58.110+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:58.383 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:58.110+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:19:58.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:58.382+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:58.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:58.382+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:58.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:58.382+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:58.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:58.383+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:19:59.346 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:59.066+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:59.346 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:59.066+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:59.346 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:59.066+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:19:59.346 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:19:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:19:59.066+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:19:59.346 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:59.266+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:59.346 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:59.266+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:59.347 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:59.266+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:19:59.347 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:19:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:19:59.267+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:19:59.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:59.346+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:59.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:59.346+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:59.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:59.346+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:19:59.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:19:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:19:59.346+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:20:00.446 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:00.222+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:00.446 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:00.222+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:00.446 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:00.222+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:00.446 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:00.223+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:20:00.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:00.349+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:00.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:00.349+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:00.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:00.349+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:00.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:00.349+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:20:00.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:00.111+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:00.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:00.111+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:00.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:00.111+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:00.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:00.112+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:20:01.388 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:01.191+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:01.388 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:01.191+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:01.388 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:01.191+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:01.388 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:01.192+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:20:01.388 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:01.074+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:01.388 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:01.074+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:01.388 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:01.074+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:01.389 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:01.075+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:20:01.389 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:20:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:20:01.090+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 318 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:20:01.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:01.388+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:01.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:01.388+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:01.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:01.388+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:01.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:01.389+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:20:02.433 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:02.169+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:02.433 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:02.169+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:02.433 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:02.169+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:02.433 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:02.170+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:20:02.433 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:02.068+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:02.434 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:02.068+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:02.434 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:02.068+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:02.434 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:02.069+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:20:02.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:02.433+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:02.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:02.433+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:02.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:02.433+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:02.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:02.434+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:20:03.446 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:03.128+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:03.446 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:03.128+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:03.446 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:03.128+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:03.446 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:03.129+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:20:03.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:03.050+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:03.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:03.050+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:03.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:03.050+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:03.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:03.051+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:20:03.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:03.444+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:03.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:03.444+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:03.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:03.444+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:03.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:03.445+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:20:04.417 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:04.138+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:04.417 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:04.138+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:04.418 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:04.138+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:04.418 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:04.139+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:20:04.418 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:04.060+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:04.418 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:04.060+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:04.418 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:04.060+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:04.418 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:04.061+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:20:04.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:04.417+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:04.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:04.417+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:04.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:04.417+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:04.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:04.418+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:20:05.371 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:05.175+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:05.371 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:05.175+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:05.371 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:05.175+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:05.372 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:05.176+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:20:05.372 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:05.029+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:05.372 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:05.029+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:05.372 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:05.029+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:05.372 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:05.030+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:20:05.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:05.437+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:05.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:05.437+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:05.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:05.437+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:05.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:05.438+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:20:06.392 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:20:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:20:06.090+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 348 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:20:06.392 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:06.225+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:06.392 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:06.225+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:06.392 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:06.225+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:06.392 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:06.225+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:20:06.392 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:06.022+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:06.393 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:06.022+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:06.393 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:06.022+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:06.393 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:06.024+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:20:06.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:06.391+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:06.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:06.391+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:06.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:06.391+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:06.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:06.392+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:20:07.236 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:06.983+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:07.236 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:06.983+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:07.236 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:06.983+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:07.236 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:06.984+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:20:07.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:07.375+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:07.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:07.375+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:07.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:07.375+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:07.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:07.375+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:20:07.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:07.236+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:07.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:07.236+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:07.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:07.236+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:07.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:07.237+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:20:08.334 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:08.023+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:08.334 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:08.023+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:08.334 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:08.023+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:08.334 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:08.024+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:20:08.335 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:08.189+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:08.335 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:08.189+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:08.335 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:08.189+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:08.335 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:08.190+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:20:08.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:08.334+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:08.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:08.334+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:08.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:08.334+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:08.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:08.335+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:20:09.350 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:09.047+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:09.350 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:09.047+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:09.350 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:09.047+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:09.350 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:09.048+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:20:09.350 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:09.175+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:09.350 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:09.175+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:09.351 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:09.175+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:09.351 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:09.176+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:20:09.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:09.349+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:09.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:09.349+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:09.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:09.349+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:09.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:09.350+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:20:10.314 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:10.139+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:10.314 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:10.139+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:10.314 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:10.139+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:10.314 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:10.140+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:20:10.315 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:10.057+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:10.315 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:10.057+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:10.315 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:10.057+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:10.315 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:10.058+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:20:10.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:10.313+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:10.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:10.313+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:10.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:10.313+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:10.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:10.314+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:20:11.312 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:20:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:20:11.090+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 376 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:20:11.312 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:11.129+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:11.312 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:11.129+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:11.312 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:11.129+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:11.312 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:11.129+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:20:11.312 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:11.033+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:11.312 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:11.033+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:11.313 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:11.033+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:11.313 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:11.034+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:20:11.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:11.311+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:11.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:11.311+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:11.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:11.311+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:11.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:11.312+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:20:12.344 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:12.012+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:12.344 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:12.012+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:12.344 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:12.012+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:12.344 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:12.013+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:20:12.344 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:12.158+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:12.344 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:12.158+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:12.344 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:12.158+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:12.344 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:12.159+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:20:12.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:12.343+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:12.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:12.343+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:12.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:12.343+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:12.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:12.344+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:20:13.319 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:13.111+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:13.319 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:13.111+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:13.319 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:13.111+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:13.320 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:13.112+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:20:13.320 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:13.035+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:13.320 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:13.035+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:13.320 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:13.035+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:13.320 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:13.035+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:20:13.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:13.318+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:13.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:13.318+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:13.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:13.318+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:13.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:13.319+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:20:14.327 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:14.124+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:14.327 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:14.124+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:14.328 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:14.124+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:14.328 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:14.125+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:20:14.328 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:13.990+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:14.328 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:13.990+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:14.328 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:13.990+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:14.328 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:13.991+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:20:14.586 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:14.327+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:14.586 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:14.327+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:14.586 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:14.327+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:14.587 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:14.328+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:20:15.331 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:14.989+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:15.331 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:14.989+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:15.331 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:14.989+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:15.331 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:14.989+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:20:15.331 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:15.174+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:15.332 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:15.174+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:15.332 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:15.174+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:15.332 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:15.175+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:20:15.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:15.330+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:15.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:15.330+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:15.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:15.330+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:15.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:15.331+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:20:16.369 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:20:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:20:16.091+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 407 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:20:16.370 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:16.221+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:16.370 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:16.221+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:16.370 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:16.221+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:16.370 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:16.222+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:20:16.370 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:15.990+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:16.370 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:15.990+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:16.370 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:15.990+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:16.370 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:15.990+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:20:16.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:16.369+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:16.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:16.369+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:16.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:16.369+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:16.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:16.370+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:20:17.338 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:17.251+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:17.338 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:17.251+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:17.338 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:17.251+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:17.338 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:17.252+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:20:17.339 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:17.016+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:17.339 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:17.016+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:17.339 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:17.016+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:17.339 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:17.016+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:20:17.659 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:17.337+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:17.659 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:17.337+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:17.659 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:17.337+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:17.659 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:17.338+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:20:18.365 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:18.033+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:18.365 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:18.033+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:18.365 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:18.033+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:18.365 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:18.034+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:20:18.365 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:18.263+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:18.366 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:18.263+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:18.366 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:18.263+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:18.366 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:18.264+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:20:18.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:18.364+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:18.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:18.364+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:18.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:18.364+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:18.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:18.365+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:20:19.287 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:18.987+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:19.287 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:18.987+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:19.287 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:18.987+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:19.288 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:18.988+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:20:19.696 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:19.287+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:19.696 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:19.287+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:19.696 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:19.287+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:19.696 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:19.288+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:20:19.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:19.391+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:19.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:19.391+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:19.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:19.391+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:19.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:19.392+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:20:20.196 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:20.029+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:20.196 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:20.029+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:20.196 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:20.029+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:20.196 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:20.030+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:20:20.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:20.347+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:20.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:20.347+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:20.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:20.347+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:20.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:20.348+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:20:20.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:20.254+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:20.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:20.254+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:20.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:20.254+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:20.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:20.255+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:20:21.394 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:20:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:20:21.091+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 434 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:20:21.395 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:21.070+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:21.395 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:21.070+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:21.395 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:21.070+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:21.395 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:21.071+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:20:21.395 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:21.263+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:21.395 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:21.263+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:21.395 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:21.263+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:21.395 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:21.264+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:20:21.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:21.394+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:21.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:21.394+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:21.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:21.394+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:21.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:21.395+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:20:22.352 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:22.245+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:22.352 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:22.245+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:22.353 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:22.245+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:22.353 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:22.245+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:20:22.353 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:22.098+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:22.353 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:22.098+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:22.353 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:22.098+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:22.353 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:22.099+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:20:22.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:22.352+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:22.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:22.352+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:22.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:22.352+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:22.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:22.353+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:20:23.390 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:23.285+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:23.390 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:23.285+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:23.390 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:23.285+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:23.391 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:23.286+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:20:23.391 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:23.086+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:23.391 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:23.086+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:23.391 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:23.086+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:23.391 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:23.087+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:20:23.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:23.389+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:23.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:23.389+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:23.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:23.389+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:23.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:23.391+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:20:24.380 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:24.092+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:24.380 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:24.092+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:24.380 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:24.092+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:24.380 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:24.093+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:20:24.380 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:24.333+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:24.380 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:24.333+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:24.380 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:24.333+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:24.380 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:24.334+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:20:24.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:24.379+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:24.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:24.379+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:24.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:24.379+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:24.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:24.380+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:20:25.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:25.349+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:25.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:25.349+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:25.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:25.349+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:25.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:25.350+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:20:25.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:25.137+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:25.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:25.137+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:25.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:25.137+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:25.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:25.138+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:20:25.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:25.309+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:25.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:25.309+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:25.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:25.309+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:25.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:25.309+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:20:26.390 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:20:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:20:26.092+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 467 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:20:26.391 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:26.129+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:26.391 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:26.129+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:26.391 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:26.129+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:26.391 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:26.130+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:20:26.392 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:26.268+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:26.392 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:26.268+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:26.392 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:26.268+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:26.392 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:26.269+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:20:26.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:26.390+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:26.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:26.390+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:26.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:26.390+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:26.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:26.391+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:20:27.392 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:27.245+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:27.393 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:27.245+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:27.393 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:27.245+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:27.393 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:27.246+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:20:27.393 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:27.122+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:27.393 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:27.122+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:27.393 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:27.122+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:27.393 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:27.123+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:20:27.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:27.392+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:27.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:27.392+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:27.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:27.392+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:27.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:27.392+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:20:28.446 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:28.286+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:28.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:28.286+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:28.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:28.286+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:28.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:28.287+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:20:28.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:28.416+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:28.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:28.416+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:28.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:28.416+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:28.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:28.417+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:20:28.449 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:28.168+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:28.449 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:28.168+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:28.449 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:28.168+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:28.449 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:28.169+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:20:29.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:29.369+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:29.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:29.369+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:29.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:29.369+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:29.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:29.369+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:20:29.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:29.335+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:29.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:29.335+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:29.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:29.335+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:29.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:29.335+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:20:29.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:29.197+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:29.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:29.198+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:29.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:29.198+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:29.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:29.199+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:20:30.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:30.398+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:30.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:30.398+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:30.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:30.398+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:30.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:30.399+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:20:30.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:30.378+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:30.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:30.378+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:30.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:30.378+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:30.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:30.379+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:20:30.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:30.214+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:30.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:30.214+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:30.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:30.214+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:30.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:30.215+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:20:31.352 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:20:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:20:31.092+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 495 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:20:31.353 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:31.235+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:31.353 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:31.235+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:31.353 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:31.235+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:31.353 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:31.235+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:20:31.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:31.447+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:31.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:31.447+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:31.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:31.447+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:31.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:31.447+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:20:31.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:31.416+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:31.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:31.416+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:31.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:31.416+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:31.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:31.417+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:20:32.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:32.439+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:32.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:32.439+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:32.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:32.439+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:32.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:32.440+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:20:32.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:32.416+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:32.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:32.416+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:32.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:32.416+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:32.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:32.417+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:20:32.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:32.240+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:32.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:32.240+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:32.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:32.240+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:32.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:32.241+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:20:33.417 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:33.258+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:33.417 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:33.258+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:33.417 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:33.258+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:33.417 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:33.259+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:20:33.696 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:33.416+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:33.696 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:33.416+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:33.696 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:33.416+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:33.696 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:33.417+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:20:33.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:33.466+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:33.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:33.466+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:33.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:33.466+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:33.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:33.467+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:20:34.597 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:34.501+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:34.597 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:34.501+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:34.598 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:34.501+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:34.598 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:34.502+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:20:34.598 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:34.371+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:34.598 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:34.371+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:34.598 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:34.371+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:34.598 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:34.372+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:20:34.599 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:34.277+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:34.599 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:34.277+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:34.599 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:34.277+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:34.599 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:34.277+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:20:35.615 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:35.535+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:35.615 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:35.535+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:35.616 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:35.535+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:35.616 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:35.536+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:20:35.616 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:35.417+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:35.616 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:35.417+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:35.616 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:35.417+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:35.616 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:35.418+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:20:35.617 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:35.294+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:35.617 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:35.294+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:35.617 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:35.294+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:35.617 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:35.295+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:20:36.339 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:20:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:20:36.093+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 525 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:20:36.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:36.487+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:36.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:36.487+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:36.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:36.487+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:36.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:36.487+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:20:36.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:36.403+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:36.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:36.403+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:36.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:36.403+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:36.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:36.404+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:20:36.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:36.339+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:36.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:36.339+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:36.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:36.339+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:36.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:36.340+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:20:37.696 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:37.445+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:37.696 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:37.446+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:37.696 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:37.446+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:37.696 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:37.447+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:20:37.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:37.292+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:37.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:37.292+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:37.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:37.292+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:37.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:37.293+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:20:37.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:37.511+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:37.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:37.512+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:37.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:37.512+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:37.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:37.513+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:20:38.550 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:38.488+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:38.550 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:38.488+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:38.550 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:38.488+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:38.550 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:38.489+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:20:38.551 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:38.262+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:38.551 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:38.262+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:38.551 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:38.262+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:38.551 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:38.263+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:20:38.946 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:38.549+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:38.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:38.549+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:38.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:38.549+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:38.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:38.551+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:20:39.531 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:39.236+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:39.531 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:39.236+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:39.531 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:39.236+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:39.531 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:39.236+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:20:39.832 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:39.576+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:39.832 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:39.576+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:39.832 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:39.576+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:39.832 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:39.577+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:20:39.832 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:39.530+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:39.832 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:39.530+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:39.832 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:39.530+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:39.832 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:39.531+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:20:40.537 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:40.276+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:40.538 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:40.276+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:40.538 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:40.276+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:40.538 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:40.277+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:20:40.946 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:40.537+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:40.946 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:40.537+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:40.946 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:40.537+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:40.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:40.538+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:20:40.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:40.539+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:40.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:40.539+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:40.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:40.539+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:40.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:40.540+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:20:41.446 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:20:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:20:41.093+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 552 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:20:41.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:41.244+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:41.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:41.244+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:41.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:41.244+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:41.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:41.245+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:20:41.949 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:41.494+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:41.949 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:41.494+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:41.949 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:41.494+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:41.949 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:41.495+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:20:41.950 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:41.536+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:41.950 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:41.536+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:41.950 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:41.536+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:41.950 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:41.536+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:20:42.510 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:42.447+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:42.511 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:42.447+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:42.511 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:42.447+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:42.511 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:42.448+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:20:42.511 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:42.203+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:42.511 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:42.203+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:42.511 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:42.203+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:42.511 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:42.203+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:20:42.946 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:42.511+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:42.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:42.511+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:42.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:42.511+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:42.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:42.511+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:20:43.521 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:43.486+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:43.521 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:43.486+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:43.521 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:43.486+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:43.521 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:43.486+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:20:43.521 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:43.465+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:43.521 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:43.465+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:43.522 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:43.465+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:43.522 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:43.466+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:20:43.522 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:43.252+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:43.522 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:43.252+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:43.522 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:43.252+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:43.522 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:43.253+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:20:44.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:44.501+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:44.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:44.501+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:44.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:44.501+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:44.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:44.502+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:20:44.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:44.276+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:44.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:44.276+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:44.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:44.276+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:44.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:44.277+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:20:44.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:44.452+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:44.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:44.452+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:44.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:44.452+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:44.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:44.453+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:20:45.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:45.531+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:45.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:45.531+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:45.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:45.531+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:45.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:45.532+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:20:45.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:45.483+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:45.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:45.483+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:45.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:45.483+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:45.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:45.484+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:20:45.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:45.304+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:45.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:45.304+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:45.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:45.304+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:45.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:45.306+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:20:46.446 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:20:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:20:46.094+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 582 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:20:46.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:46.288+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:46.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:46.288+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:46.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:46.288+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:46.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:46.289+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:20:46.946 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:46.559+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:46.946 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:46.559+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:46.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:46.559+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:46.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:46.560+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:20:46.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:46.493+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:46.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:46.494+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:46.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:46.494+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:46.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:46.495+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:20:47.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:47.558+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:47.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:47.558+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:47.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:47.558+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:47.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:47.559+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:20:47.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:47.528+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:47.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:47.528+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:47.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:47.528+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:47.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:47.528+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:20:47.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:47.326+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:47.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:47.326+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:47.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:47.326+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:47.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:47.327+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:20:48.561 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:48.302+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:48.562 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:48.302+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:48.562 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:48.302+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:48.562 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:48.303+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:20:48.946 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:48.585+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:48.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:48.585+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:48.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:48.585+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:48.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:48.587+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:20:48.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:48.561+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:48.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:48.561+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:48.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:48.561+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:48.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:48.562+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:20:49.599 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:49.521+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:49.600 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:49.521+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:49.600 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:49.521+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:49.600 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:49.522+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:20:49.600 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:49.275+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:49.600 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:49.275+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:49.600 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:49.275+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:49.600 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:49.276+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:20:49.946 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:49.599+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:49.946 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:49.599+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:49.946 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:49.599+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:49.946 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:49.600+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:20:50.557 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:50.237+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:50.557 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:50.237+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:50.557 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:50.237+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:50.557 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:50.238+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:20:50.946 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:50.620+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:50.946 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:50.620+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:50.946 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:50.620+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:50.946 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:50.621+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:20:50.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:50.557+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:50.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:50.557+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:50.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:50.557+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:50.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:50.557+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:20:51.446 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:20:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:20:51.094+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 609 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:20:51.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:51.233+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:51.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:51.233+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:51.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:51.233+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:51.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:51.234+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:20:51.946 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:51.667+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:51.946 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:51.667+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:51.946 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:51.667+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:51.946 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:51.668+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:20:51.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:51.547+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:51.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:51.547+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:51.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:51.547+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:51.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:51.548+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:20:52.568 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:52.276+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:52.568 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:52.276+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:52.569 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:52.276+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:52.569 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:52.277+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:20:52.946 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:52.656+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:52.946 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:52.656+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:52.946 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:52.656+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:52.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:52.657+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:20:52.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:52.568+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:52.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:52.568+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:52.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:52.568+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:52.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:52.568+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:20:53.521 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:53.258+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:53.521 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:53.258+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:53.521 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:53.258+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:53.521 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:53.259+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:20:53.946 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:53.681+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:53.946 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:53.681+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:53.946 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:53.681+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:53.946 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:53.682+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:20:53.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:53.586+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:53.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:53.586+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:53.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:53.586+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:53.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:53.587+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:20:54.637 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:54.277+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:54.637 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:54.277+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:54.637 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:54.277+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:54.637 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:54.278+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:20:54.946 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:54.636+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:54.946 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:54.636+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:54.946 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:54.636+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:54.946 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:54.636+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:20:54.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:54.636+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:54.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:54.636+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:54.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:54.636+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:54.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:54.637+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:20:55.635 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:55.265+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:55.635 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:55.265+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:55.635 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:55.265+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:55.635 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:55.265+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:20:55.946 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:55.674+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:55.946 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:55.674+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:55.946 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:55.674+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:55.946 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:55.675+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:20:55.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:55.634+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:55.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:55.634+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:55.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:55.634+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:55.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:55.635+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:20:56.446 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:20:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:20:56.094+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 639 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:20:56.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:56.241+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:56.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:56.241+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:56.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:56.241+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:56.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:56.242+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:20:56.946 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:56.679+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:56.946 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:56.679+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:56.946 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:56.679+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:56.946 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:56.680+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:20:56.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:56.602+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:56.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:56.603+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:56.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:56.603+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:56.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:56.604+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:20:57.626 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:57.205+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:57.626 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:57.205+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:57.626 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:57.205+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:57.626 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:57.206+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:20:57.946 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:57.664+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:57.946 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:57.664+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:57.946 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:57.664+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:57.946 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:57.665+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:20:57.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:57.625+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:57.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:57.625+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:57.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:57.625+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:57.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:57.626+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:20:58.604 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:58.226+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:58.604 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:58.226+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:58.605 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:58.226+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:58.605 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:58.227+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:20:58.946 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:58.692+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:58.946 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:58.692+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:58.946 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:58.692+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:58.946 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:58.692+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:20:58.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:58.603+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:58.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:58.604+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:58.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:58.604+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:58.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:58.604+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:20:59.593 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:59.208+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:59.593 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:59.208+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:59.593 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:59.208+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:20:59.593 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:20:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:20:59.209+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:20:59.946 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:59.728+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:59.946 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:59.728+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:59.946 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:59.728+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:20:59.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:20:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:20:59.729+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:20:59.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:59.592+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:59.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:59.592+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:59.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:59.592+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:20:59.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:20:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:20:59.593+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:21:00.626 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:00.223+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:00.626 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:00.224+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:00.626 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:00.224+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:00.626 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:00.224+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:21:00.946 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:00.778+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:00.946 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:00.778+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:00.946 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:00.778+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:00.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:00.779+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:21:00.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:00.625+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:00.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:00.625+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:00.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:00.625+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:00.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:00.626+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:21:01.447 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:21:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:21:01.095+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 666 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:21:01.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:01.179+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:01.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:01.179+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:01.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:01.179+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:01.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:01.180+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:21:01.946 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:01.787+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:01.946 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:01.787+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:01.946 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:01.787+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:01.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:01.788+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:21:01.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:01.671+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:01.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:01.671+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:01.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:01.671+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:01.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:01.672+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:21:02.696 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:02.210+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:02.696 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:02.210+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:02.696 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:02.210+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:02.696 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:02.211+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:21:03.183 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:02.791+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:03.183 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:02.791+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:03.183 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:02.791+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:03.183 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:02.792+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:21:03.183 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:02.713+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:03.183 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:02.713+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:03.183 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:02.713+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:03.183 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:02.714+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:21:03.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:03.182+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:03.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:03.182+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:03.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:03.182+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:03.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:03.183+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:21:03.946 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:03.744+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:03.946 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:03.744+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:03.946 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:03.744+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:03.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:03.745+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:21:03.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:03.716+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:03.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:03.716+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:03.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:03.716+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:03.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:03.716+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:21:04.696 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:04.224+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:04.696 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:04.224+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:04.696 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:04.224+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:04.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:04.225+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:21:05.196 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:04.743+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:05.196 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:04.743+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:05.196 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:04.743+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:05.196 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:04.744+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:21:05.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:04.745+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:05.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:04.745+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:05.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:04.745+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:05.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:04.746+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:21:05.696 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:05.218+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:05.696 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:05.218+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:05.696 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:05.218+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:05.696 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:05.219+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:21:06.098 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:05.733+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:06.098 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:05.733+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:06.098 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:05.733+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:06.098 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:05.733+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:21:06.098 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:05.792+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:06.098 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:05.792+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:06.099 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:05.792+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:06.099 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:05.792+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:21:06.446 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:21:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:21:06.096+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 697 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:21:06.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:06.253+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:06.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:06.253+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:06.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:06.253+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:06.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:06.253+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:21:07.196 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:06.777+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:07.196 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:06.777+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:07.196 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:06.777+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:07.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:06.778+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:21:07.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:06.735+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:07.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:06.735+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:07.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:06.735+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:07.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:06.736+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:21:07.696 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:07.277+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:07.696 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:07.277+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:07.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:07.277+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:07.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:07.278+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:21:08.196 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:07.776+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:08.196 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:07.776+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:08.196 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:07.776+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:08.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:07.777+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:21:08.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:07.720+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:08.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:07.720+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:08.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:07.720+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:08.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:07.721+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:21:08.696 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:08.323+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:08.696 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:08.323+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:08.696 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:08.323+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:08.696 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:08.324+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:21:09.196 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:08.754+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:09.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:08.754+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:09.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:08.754+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:09.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:08.754+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:21:09.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:08.703+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:09.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:08.703+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:09.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:08.703+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:09.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:08.704+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:21:09.660 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:09.278+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:09.660 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:09.278+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:09.660 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:09.278+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:09.660 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:09.279+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:21:09.946 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:09.731+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:09.946 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:09.731+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:09.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:09.731+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:09.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:09.732+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:21:09.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:09.659+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:09.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:09.659+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:09.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:09.659+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:09.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:09.660+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:21:10.688 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:10.288+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:10.688 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:10.288+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:10.688 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:10.288+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:10.688 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:10.288+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:21:10.946 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:10.747+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:10.946 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:10.747+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:10.946 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:10.747+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:10.946 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:10.748+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:21:10.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:10.687+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:10.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:10.687+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:10.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:10.687+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:10.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:10.688+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:21:11.446 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:21:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:21:11.096+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 724 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:21:11.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:11.266+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:11.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:11.266+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:11.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:11.266+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:11.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:11.266+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:21:12.196 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:11.779+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:12.196 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:11.779+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:12.196 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:11.779+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:12.196 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:11.779+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:21:12.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:11.735+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:12.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:11.735+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:12.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:11.735+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:12.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:11.736+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:21:12.537 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:12.228+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:12.538 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:12.228+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:12.538 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:12.228+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:12.538 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:12.228+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:21:12.946 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:12.764+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:12.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:12.764+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:12.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:12.764+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:12.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:12.765+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:21:12.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:12.771+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:12.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:12.771+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:12.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:12.771+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:12.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:12.772+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:21:13.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:13.193+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:13.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:13.193+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:13.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:13.193+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:13.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:13.194+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:21:14.190 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:13.813+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:14.190 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:13.813+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:14.190 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:13.813+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:14.191 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:13.814+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:21:14.191 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:13.764+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:14.191 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:13.764+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:14.191 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:13.764+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:14.191 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:13.765+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:21:14.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:14.189+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:14.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:14.189+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:14.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:14.189+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:14.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:14.190+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:21:15.171 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:14.765+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:15.172 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:14.765+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:15.172 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:14.765+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:15.172 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:14.766+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:21:15.172 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:14.814+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:15.172 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:14.814+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:15.172 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:14.814+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:15.172 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:14.815+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:21:15.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:15.170+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:15.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:15.170+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:15.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:15.171+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:15.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:15.172+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:21:16.026 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:15.830+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:16.027 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:15.830+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:16.027 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:15.830+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:16.027 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:15.831+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:21:16.027 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:15.766+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:16.027 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:15.766+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:16.027 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:15.766+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:16.027 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:15.768+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:21:16.446 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:21:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:21:16.097+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 769 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:21:16.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:16.162+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:16.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:16.162+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:16.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:16.162+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:16.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:16.163+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:21:17.142 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:16.822+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:17.142 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:16.822+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:17.142 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:16.822+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:17.142 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:16.823+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:21:17.142 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:16.726+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:17.143 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:16.726+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:17.143 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:16.726+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:17.143 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:16.727+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:21:17.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:17.141+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:17.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:17.141+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:17.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:17.141+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:17.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:17.142+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:21:18.099 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:17.809+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:18.100 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:17.809+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:18.100 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:17.809+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:18.100 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:17.810+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:21:18.100 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:17.699+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:18.100 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:17.699+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:18.100 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:17.699+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:18.100 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:17.700+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:21:18.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:18.099+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:18.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:18.099+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:18.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:18.099+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:18.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:18.100+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:21:18.946 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:18.682+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:18.946 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:18.682+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:18.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:18.682+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:18.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:18.683+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:21:18.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:18.779+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:18.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:18.779+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:18.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:18.779+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:18.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:18.779+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:21:19.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:19.130+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:19.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:19.130+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:19.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:19.130+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:19.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:19.132+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:21:20.162 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:19.717+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:20.163 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:19.717+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:20.163 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:19.717+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:20.163 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:19.718+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:21:20.163 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:19.791+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:20.163 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:19.791+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:20.163 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:19.791+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:20.163 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:19.792+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:21:20.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:20.162+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:20.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:20.162+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:20.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:20.162+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:20.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:20.162+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:21:21.098 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:20.744+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:21.098 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:20.744+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:21.098 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:20.744+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:21.098 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:20.745+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:21:21.099 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:20.762+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:21.099 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:20.762+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:21.099 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:20.762+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:21.099 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:20.763+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:21:21.446 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:21:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:21:21.097+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 796 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:21:21.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:21.172+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:21.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:21.172+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:21.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:21.172+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:21.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:21.173+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:21:22.133 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:21.753+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:22.134 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:21.753+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:22.134 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:21.753+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:22.134 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:21.753+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:21:22.134 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:21.772+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:22.134 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:21.772+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:22.134 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:21.772+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:22.134 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:21.772+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:21:22.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:22.133+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:22.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:22.133+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:22.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:22.133+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:22.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:22.134+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:21:23.091 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:22.756+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:23.091 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:22.756+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:23.091 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:22.756+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:23.091 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:22.757+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:21:23.092 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:22.775+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:23.092 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:22.775+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:23.092 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:22.775+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:23.092 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:22.776+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:21:23.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:23.088+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:23.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:23.088+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:23.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:23.088+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:23.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:23.089+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:21:24.105 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:23.798+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:24.105 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:23.798+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:24.105 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:23.798+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:24.105 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:23.799+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:21:24.105 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:23.800+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:24.105 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:23.800+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:24.105 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:23.800+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:24.105 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:23.801+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:21:24.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:24.104+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:24.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:24.104+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:24.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:24.104+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:24.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:24.105+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:21:25.077 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:24.827+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:25.078 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:24.827+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:25.078 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:24.827+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:25.078 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:24.828+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:21:25.078 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:24.795+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:25.078 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:24.796+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:25.078 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:24.796+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:25.078 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:24.797+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:21:25.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:25.077+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:25.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:25.077+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:25.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:25.077+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:25.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:25.077+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:21:26.098 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:25.763+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:26.099 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:25.763+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:26.099 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:25.763+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:26.099 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:25.764+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:21:26.099 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:25.862+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:26.100 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:25.862+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:26.100 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:25.862+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:26.100 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:25.863+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:21:26.446 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:21:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:21:26.098+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 826 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:21:26.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:26.121+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:26.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:26.121+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:26.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:26.121+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:26.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:26.122+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:21:27.169 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:26.841+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:27.169 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:26.841+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:27.170 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:26.841+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:27.170 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:26.841+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:21:27.170 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:26.786+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:27.170 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:26.786+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:27.170 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:26.786+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:27.170 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:26.787+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:21:27.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:27.168+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:27.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:27.168+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:27.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:27.168+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:27.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:27.169+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:21:28.143 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:27.830+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:28.143 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:27.830+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:28.143 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:27.830+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:28.143 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:27.831+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:21:28.143 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:27.794+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:28.143 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:27.794+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:28.143 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:27.794+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:28.144 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:27.795+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:21:28.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:28.143+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:28.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:28.143+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:28.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:28.143+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:28.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:28.143+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:21:29.176 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:28.831+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:29.176 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:28.831+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:29.176 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:28.831+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:29.176 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:28.832+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:21:29.176 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:28.872+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:29.176 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:28.873+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:29.177 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:28.873+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:29.177 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:28.873+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:21:29.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:29.176+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:29.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:29.176+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:29.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:29.176+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:29.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:29.177+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:21:30.150 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:29.878+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:30.150 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:29.878+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:30.151 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:29.878+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:30.151 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:29.879+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:21:30.151 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:29.827+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:30.151 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:29.827+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:30.151 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:29.827+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:30.151 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:29.827+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:21:30.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:30.151+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:30.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:30.151+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:30.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:30.151+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:30.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:30.152+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:21:31.107 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:30.853+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:31.107 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:30.853+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:31.107 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:30.853+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:31.107 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:30.854+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:21:31.107 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:30.875+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:31.107 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:30.875+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:31.107 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:30.875+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:31.108 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:30.876+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:21:31.108 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:21:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:21:31.099+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 853 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:21:31.388 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:31.196+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:31.388 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:31.196+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:31.388 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:31.196+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:31.388 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:31.197+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:21:32.157 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:31.880+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:32.157 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:31.880+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:32.158 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:31.880+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:32.158 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:31.881+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:21:32.158 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:31.832+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:32.158 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:31.832+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:32.158 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:31.832+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:32.158 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:31.833+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:21:32.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:32.157+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:32.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:32.157+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:32.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:32.157+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:32.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:32.158+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:21:33.120 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:32.814+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:33.120 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:32.814+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:33.120 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:32.814+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:33.120 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:32.815+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:21:33.121 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:32.905+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:33.121 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:32.905+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:33.121 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:32.905+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:33.121 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:32.906+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:21:33.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:33.119+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:33.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:33.119+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:33.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:33.119+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:33.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:33.120+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:21:34.196 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:33.910+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:34.196 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:33.910+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:34.196 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:33.910+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:34.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:33.911+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:21:34.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:33.850+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:34.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:33.850+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:34.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:33.851+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:34.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:33.852+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:21:34.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:34.100+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:34.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:34.100+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:34.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:34.100+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:34.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:34.100+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:21:35.149 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:34.944+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:35.149 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:34.944+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:35.150 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:34.944+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:35.150 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:34.944+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:21:35.150 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:34.832+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:35.150 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:34.832+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:35.150 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:34.832+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:35.150 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:34.833+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:21:35.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:35.148+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:35.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:35.148+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:35.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:35.148+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:35.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:35.150+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:21:36.100 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:35.825+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:36.100 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:35.825+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:36.100 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:35.825+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:36.100 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:35.826+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:21:36.101 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:35.979+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:36.101 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:35.979+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:36.101 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:35.979+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:36.101 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:35.980+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:21:36.446 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:21:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:21:36.099+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 883 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:21:36.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:36.114+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:36.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:36.114+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:36.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:36.114+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:36.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:36.115+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:21:37.090 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:36.819+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:37.090 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:36.819+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:37.090 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:36.819+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:37.090 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:36.820+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:21:37.090 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:36.985+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:37.090 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:36.985+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:37.091 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:36.985+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:37.091 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:36.986+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:21:37.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:37.089+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:37.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:37.089+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:37.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:37.089+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:37.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:37.090+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:21:38.102 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:37.817+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:38.102 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:37.817+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:38.102 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:37.817+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:38.102 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:37.818+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:21:38.102 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:38.006+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:38.102 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:38.006+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:38.102 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:38.006+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:38.102 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:38.007+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:21:38.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:38.101+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:38.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:38.101+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:38.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:38.101+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:38.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:38.102+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:21:39.143 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:38.982+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:39.143 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:38.982+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:39.143 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:38.982+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:39.144 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:38.983+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:21:39.144 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:38.845+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:39.144 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:38.845+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:39.144 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:38.845+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:39.144 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:38.846+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:21:39.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:39.142+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:39.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:39.142+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:39.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:39.142+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:39.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:39.143+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:21:40.110 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:39.942+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:40.110 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:39.942+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:40.110 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:39.942+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:40.110 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:39.943+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:21:40.110 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:39.797+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:40.110 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:39.797+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:40.110 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:39.797+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:40.111 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:39.798+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:21:40.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:40.109+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:40.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:40.109+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:40.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:40.109+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:40.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:40.110+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:21:41.100 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:40.965+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:41.101 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:40.965+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:41.101 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:40.965+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:41.101 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:40.966+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:21:41.101 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:40.832+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:41.101 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:40.833+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:41.101 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:40.833+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:41.101 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:40.834+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:21:41.102 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:41.071+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:41.102 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:41.071+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:41.102 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:41.071+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:41.102 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:41.072+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:21:41.446 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:21:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:21:41.099+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 910 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:21:42.196 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:41.990+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:42.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:41.990+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:42.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:41.990+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:42.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:41.990+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:21:42.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:42.119+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:42.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:42.119+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:42.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:42.119+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:42.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:42.120+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:21:42.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:41.873+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:42.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:41.873+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:42.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:41.873+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:42.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:41.874+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:21:43.147 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:43.030+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:43.148 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:43.030+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:43.148 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:43.030+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:43.148 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:43.031+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:21:43.148 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:42.865+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:43.148 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:42.865+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:43.148 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:42.865+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:43.148 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:42.866+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:21:43.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:43.147+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:43.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:43.147+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:43.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:43.147+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:43.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:43.148+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:21:44.123 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:44.059+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:44.123 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:44.059+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:44.124 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:44.059+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:44.124 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:44.059+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:21:44.124 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:43.823+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:44.124 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:43.823+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:44.124 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:43.823+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:44.124 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:43.824+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:21:44.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:44.122+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:44.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:44.122+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:44.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:44.122+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:44.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:44.123+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:21:45.145 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:45.063+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:45.145 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:45.063+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:45.145 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:45.063+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:45.145 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:45.065+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:21:45.145 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:44.825+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:45.145 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:44.825+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:45.145 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:44.825+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:45.145 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:44.826+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:21:45.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:45.143+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:45.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:45.144+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:45.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:45.144+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:45.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:45.145+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:21:46.101 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:45.837+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:46.101 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:45.837+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:46.101 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:45.837+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:46.101 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:45.838+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:21:46.101 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:46.035+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:46.102 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:46.036+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:46.102 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:46.036+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:46.102 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:46.037+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:21:46.446 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:21:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:21:46.100+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 940 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:21:46.455 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:46.158+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:46.455 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:46.158+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:46.455 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:46.158+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:46.455 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:46.159+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:21:47.129 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:46.828+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:47.130 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:46.829+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:47.130 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:46.829+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:47.130 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:46.829+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:21:47.130 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:47.048+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:47.130 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:47.048+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:47.130 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:47.048+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:47.131 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:47.049+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:21:47.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:47.128+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:47.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:47.128+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:47.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:47.128+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:47.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:47.129+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:21:48.053 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:47.796+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:48.054 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:47.796+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:48.054 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:47.796+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:48.054 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:47.797+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:21:48.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:48.133+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:48.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:48.133+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:48.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:48.133+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:48.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:48.135+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:21:48.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:48.053+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:48.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:48.053+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:48.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:48.053+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:48.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:48.054+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:21:49.105 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:48.830+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:49.106 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:48.830+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:49.106 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:48.830+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:49.106 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:48.831+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:21:49.106 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:49.047+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:49.106 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:49.047+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:49.106 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:49.047+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:49.106 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:49.047+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:21:49.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:49.105+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:49.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:49.105+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:49.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:49.105+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:49.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:49.105+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:21:50.146 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:49.831+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:50.147 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:49.831+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:50.147 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:49.831+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:50.147 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:49.832+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:21:50.147 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:50.038+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:50.147 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:50.038+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:50.147 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:50.038+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:50.147 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:50.039+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:21:50.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:50.146+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:50.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:50.146+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:50.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:50.146+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:50.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:50.147+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:21:51.112 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:21:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:21:51.100+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 967 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:21:51.112 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:51.022+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:51.112 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:51.022+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:51.113 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:51.022+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:51.113 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:51.023+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:21:51.113 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:50.853+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:51.113 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:50.853+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:51.113 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:50.853+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:51.113 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:50.854+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:21:51.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:51.142+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:51.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:51.142+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:51.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:51.142+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:51.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:51.143+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:21:52.186 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:51.885+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:52.187 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:51.885+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:52.187 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:51.885+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:52.187 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:51.886+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:21:52.187 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:52.026+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:52.187 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:52.026+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:52.187 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:52.026+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:52.187 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:52.027+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:21:52.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:52.185+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:52.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:52.185+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:52.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:52.185+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:52.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:52.186+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:21:53.145 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:53.056+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:53.146 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:53.056+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:53.146 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:53.056+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:53.146 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:53.057+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:21:53.146 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:52.840+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:53.146 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:52.840+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:53.146 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:52.840+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:53.146 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:52.841+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:21:53.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:53.145+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:53.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:53.145+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:53.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:53.145+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:53.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:53.145+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:21:54.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:54.109+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:54.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:54.109+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:54.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:54.109+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:54.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:54.110+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:21:54.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:53.872+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:54.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:53.872+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:54.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:53.872+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:54.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:53.873+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:21:54.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:54.022+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:54.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:54.022+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:54.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:54.022+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:54.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:54.023+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:21:55.153 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:54.889+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:55.153 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:54.889+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:55.153 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:54.889+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:55.153 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:54.890+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:21:55.154 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:54.977+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:55.154 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:54.977+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:55.154 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:54.977+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:55.154 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:54.978+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:21:55.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:55.152+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:55.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:55.152+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:55.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:55.152+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:55.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:55.153+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:21:56.196 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:55.991+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:56.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:55.991+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:56.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:55.991+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:56.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:55.992+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:21:56.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:55.906+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:56.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:55.906+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:56.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:55.906+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:56.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:55.907+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:21:56.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:56.140+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:56.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:56.140+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:56.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:56.140+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:56.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:56.141+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:21:56.198 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:21:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:21:56.101+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 1012 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:21:57.196 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:56.876+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:57.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:56.876+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:57.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:56.876+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:57.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:56.877+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:21:57.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:56.965+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:57.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:56.965+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:57.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:56.965+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:57.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:56.965+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:21:57.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:57.109+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:57.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:57.109+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:57.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:57.109+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:57.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:57.110+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:21:58.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:57.920+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:58.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:57.920+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:58.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:57.920+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:58.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:57.921+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:21:58.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:58.072+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:58.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:58.072+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:58.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:58.072+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:58.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:58.073+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:21:58.199 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:58.008+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:58.199 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:58.008+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:58.199 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:58.008+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:58.199 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:58.009+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:21:59.196 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:58.965+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:59.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:58.965+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:59.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:58.965+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:21:59.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:58.967+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:21:59.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:58.887+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:59.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:58.887+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:59.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:58.887+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:21:59.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:58.888+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:21:59.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:59.056+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:59.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:59.056+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:59.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:59.056+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:21:59.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:21:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:21:59.057+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:22:00.196 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:59.961+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:00.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:59.961+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:00.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:59.961+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:00.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:21:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:21:59.961+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:22:00.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:59.874+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:00.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:59.874+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:00.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:59.874+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:00.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:21:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:21:59.875+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:22:00.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:00.060+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:00.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:00.060+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:00.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:00.060+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:00.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:00.061+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:22:01.196 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:22:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:22:01.101+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 1057 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:22:01.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:00.939+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:01.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:00.939+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:01.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:00.939+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:01.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:00.940+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:22:01.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:01.057+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:01.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:01.057+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:01.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:01.057+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:01.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:01.058+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:22:01.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:00.875+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:01.199 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:00.875+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:01.199 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:00.875+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:01.199 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:00.876+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:22:02.107 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:01.962+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:02.107 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:01.962+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:02.107 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:01.962+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:02.107 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:01.963+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:22:02.108 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:01.838+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:02.108 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:01.838+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:02.108 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:01.838+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:02.108 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:01.839+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:22:02.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:02.105+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:02.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:02.105+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:02.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:02.105+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:02.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:02.107+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:22:03.196 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:02.914+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:03.196 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:02.914+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:03.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:02.914+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:03.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:02.914+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:22:03.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:02.819+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:03.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:02.819+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:03.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:02.819+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:03.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:02.820+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:22:03.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:03.066+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:03.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:03.066+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:03.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:03.066+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:03.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:03.067+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:22:04.101 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:03.867+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:04.101 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:03.867+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:04.101 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:03.867+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:04.101 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:03.868+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:22:04.101 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:03.811+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:04.102 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:03.811+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:04.102 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:03.811+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:04.102 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:03.812+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:22:04.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:04.100+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:04.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:04.100+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:04.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:04.100+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:04.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:04.101+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:22:05.083 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:04.847+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:05.083 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:04.847+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:05.083 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:04.847+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:05.083 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:04.848+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:22:05.083 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:04.828+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:05.083 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:04.828+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:05.083 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:04.828+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:05.084 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:04.829+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:22:05.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:05.082+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:05.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:05.082+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:05.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:05.082+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:05.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:05.083+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:22:06.098 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:05.844+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:06.098 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:05.844+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:06.098 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:05.844+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:06.098 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:05.844+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:22:06.099 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:05.790+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:06.099 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:05.790+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:06.099 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:05.790+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:06.099 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:05.790+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:22:06.446 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:22:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:22:06.102+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 1104 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:22:06.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:06.097+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:06.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:06.097+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:06.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:06.097+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:06.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:06.098+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:22:07.118 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:06.836+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:07.118 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:06.836+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:07.118 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:06.836+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:07.118 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:06.837+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:22:07.119 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:06.851+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:07.119 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:06.851+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:07.119 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:06.851+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:07.119 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:06.852+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:22:07.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:07.117+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:07.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:07.117+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:07.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:07.117+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:07.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:07.118+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:22:08.148 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:07.884+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:08.148 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:07.884+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:08.148 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:07.884+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:08.148 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:07.885+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:22:08.148 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:07.822+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:08.148 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:07.822+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:08.148 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:07.822+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:08.148 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:07.822+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:22:08.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:08.147+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:08.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:08.147+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:08.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:08.147+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:08.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:08.147+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:22:09.173 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:08.902+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:09.173 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:08.902+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:09.174 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:08.902+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:09.174 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:08.902+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:22:09.174 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:08.858+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:09.174 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:08.858+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:09.174 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:08.858+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:09.174 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:08.858+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:22:09.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:09.172+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:09.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:09.172+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:09.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:09.172+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:09.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:09.174+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:22:10.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:09.936+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:10.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:09.936+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:10.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:09.936+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:10.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:09.937+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:22:10.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:09.837+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:10.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:09.837+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:10.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:09.837+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:10.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:09.838+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:22:10.696 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:10.209+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:10.696 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:10.209+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:10.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:10.209+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:10.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:10.209+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:22:11.103 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:10.843+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:11.103 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:10.843+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:11.104 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:10.843+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:11.104 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:10.844+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:22:11.104 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:10.974+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:11.104 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:10.974+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:11.104 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:10.974+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:11.104 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:10.975+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:22:11.446 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:22:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:22:11.103+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 1148 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:22:11.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:11.253+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:11.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:11.253+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:11.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:11.253+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:11.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:11.254+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:22:12.196 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:12.021+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:12.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:12.021+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:12.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:12.021+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:12.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:12.023+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:22:12.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:11.850+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:12.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:11.850+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:12.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:11.850+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:12.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:11.851+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:22:12.696 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:12.291+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:12.696 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:12.291+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:12.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:12.291+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:12.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:12.292+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:22:13.196 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:12.877+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:13.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:12.877+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:13.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:12.877+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:13.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:12.878+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:22:13.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:13.019+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:13.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:13.019+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:13.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:13.019+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:13.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:13.020+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:22:13.696 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:13.308+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:13.696 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:13.308+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:13.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:13.308+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:13.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:13.310+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:22:14.196 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:14.053+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:14.196 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:14.053+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:14.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:14.053+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:14.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:14.053+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:22:14.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:13.852+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:14.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:13.852+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:14.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:13.852+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:14.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:13.853+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:22:14.696 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:14.337+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:14.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:14.337+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:14.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:14.337+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:14.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:14.337+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:22:15.196 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:15.014+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:15.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:15.014+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:15.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:15.014+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:15.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:15.015+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:22:15.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:14.834+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:15.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:14.834+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:15.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:14.834+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:15.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:14.835+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:22:15.696 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:15.384+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:15.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:15.384+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:15.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:15.384+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:15.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:15.385+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:22:16.104 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:15.821+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:16.104 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:15.821+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:16.104 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:15.821+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:16.104 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:15.821+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:22:16.105 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:16.031+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:16.105 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:16.031+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:16.105 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:16.031+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:16.105 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:16.032+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:22:16.446 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:22:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:22:16.104+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 1195 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:22:16.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:16.342+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:16.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:16.342+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:16.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:16.342+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:16.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:16.343+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:22:17.196 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:17.060+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:17.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:17.060+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:17.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:17.060+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:17.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:17.061+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:22:17.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:16.824+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:17.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:16.824+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:17.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:16.824+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:17.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:16.825+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:22:17.696 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:17.300+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:17.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:17.300+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:17.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:17.300+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:17.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:17.301+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:22:18.196 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:17.830+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:18.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:17.830+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:18.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:17.830+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:18.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:17.831+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:22:18.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:18.062+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:18.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:18.062+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:18.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:18.062+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:18.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:18.063+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:22:18.306 INFO:teuthology.orchestra.run.smithi079.stderr:2024-05-04T12:22:18.303+0000 7ffb414f7700 0 monclient(hunting): authenticate timed out after 300 2024-05-04T12:22:18.306 INFO:teuthology.orchestra.run.smithi079.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2024-05-04T12:22:18.662 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:18.273+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:18.663 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:18.273+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:18.663 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:18.273+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:18.663 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:18.273+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:22:18.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:18.831+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:18.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:18.831+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:18.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:18.831+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:18.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:18.832+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:22:19.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-04T12:22:19.072 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/task/parallel.py", line 56, in task p.spawn(_run_spawned, ctx, confg, taskname) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/task/parallel.py", line 64, in _run_spawned mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/task/sequential.py", line 47, in task mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=confg) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_09dbd6bba52d19ba471c33bc15c008e6ad158ea6/qa/tasks/cephadm.py", line 1469, in shell _shell( File "/home/teuthworker/src/github.com_ceph_ceph-c_09dbd6bba52d19ba471c33bc15c008e6ad158ea6/qa/tasks/cephadm.py", line 110, in _shell return remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi079 with status 1: "sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d9b985c4-0a0e-11ef-bc97-c7b262605968 -e sha1=09dbd6bba52d19ba471c33bc15c008e6ad158ea6 -- bash -c 'ceph orch ps'" 2024-05-04T12:22:19.304 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=3eaf4662403647c6bbbc16e6b83a74fb Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/task/parallel.py", line 56, in task p.spawn(_run_spawned, ctx, confg, taskname) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/task/parallel.py", line 64, in _run_spawned mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/task/sequential.py", line 47, in task mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=confg) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_09dbd6bba52d19ba471c33bc15c008e6ad158ea6/qa/tasks/cephadm.py", line 1469, in shell _shell( File "/home/teuthworker/src/github.com_ceph_ceph-c_09dbd6bba52d19ba471c33bc15c008e6ad158ea6/qa/tasks/cephadm.py", line 110, in _shell return remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi079 with status 1: "sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d9b985c4-0a0e-11ef-bc97-c7b262605968 -e sha1=09dbd6bba52d19ba471c33bc15c008e6ad158ea6 -- bash -c 'ceph orch ps'" 2024-05-04T12:22:19.308 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-05-04T12:22:19.319 INFO:tasks.kclient:Unmounting kernel clients... 2024-05-04T12:22:19.320 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-05-04T12:22:19.320 DEBUG:teuthology.orchestra.run.smithi079:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-04T12:22:19.342 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-05-04T12:22:19.342 DEBUG:teuthology.orchestra.run.smithi079:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-04T12:22:19.398 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-04T12:22:19.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:19.099+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:19.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:19.099+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:19.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:19.099+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:19.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:19.100+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:22:19.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:19.258+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:19.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:19.258+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:19.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:19.258+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:19.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:19.259+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:22:20.070 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:19.813+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:20.070 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:19.813+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:20.070 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:19.813+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:20.070 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:19.814+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:22:20.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:20.069+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:20.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:20.069+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:20.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:20.069+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:20.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:20.070+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:22:20.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:20.253+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:20.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:20.253+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:20.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:20.253+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:20.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:20.254+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:22:21.105 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:20.844+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:21.105 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:20.844+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:21.105 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:20.844+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:21.105 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:20.845+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:22:21.446 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:22:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:22:21.104+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 1239 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:22:21.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:21.267+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:21.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:21.267+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:21.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:21.267+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:21.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:21.268+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:22:21.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:21.115+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:21.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:21.115+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:21.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:21.115+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:21.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:21.116+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:22:22.164 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:21.832+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:22.164 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:21.832+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:22.164 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:21.832+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:22.165 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:21.833+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:22:22.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:22.163+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:22.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:22.163+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:22.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:22.163+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:22.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:22.164+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:22:22.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:22.311+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:22.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:22.311+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:22.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:22.311+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:22.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:22.312+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:22:23.180 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:22.863+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:23.180 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:22.863+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:23.181 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:22.863+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:23.181 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:22.864+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:22:23.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:23.321+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:23.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:23.321+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:23.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:23.321+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:23.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:23.322+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:22:23.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:23.179+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:23.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:23.179+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:23.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:23.179+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:23.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:23.181+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:22:24.196 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:23.871+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:24.196 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:23.871+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:24.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:23.871+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:24.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:23.872+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:22:24.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:24.206+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:24.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:24.206+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:24.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:24.206+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:24.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:24.207+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:22:24.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:24.286+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:24.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:24.286+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:24.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:24.286+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:24.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:24.286+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:22:25.196 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:24.889+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:25.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:24.889+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:25.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:24.889+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:25.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:24.890+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:22:25.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:25.223+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:25.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:25.223+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:25.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:25.223+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:25.699 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:25.224+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:22:25.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:25.319+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:25.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:25.319+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:25.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:25.319+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:25.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:25.320+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:22:26.105 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:25.848+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:26.105 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:25.848+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:26.106 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:25.848+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:26.106 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:25.849+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:22:26.446 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:22:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:22:26.104+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 1344 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:22:26.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:26.205+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:26.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:26.205+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:26.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:26.205+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:26.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:26.206+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:22:26.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:26.283+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:26.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:26.283+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:26.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:26.283+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:26.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:26.284+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:22:27.196 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:26.821+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:27.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:26.821+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:27.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:26.821+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:27.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:26.822+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:22:27.564 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:27.289+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:27.565 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:27.289+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:27.565 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:27.289+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:27.565 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:27.290+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:22:27.565 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:27.237+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:27.565 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:27.237+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:27.565 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:27.237+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:27.565 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:27.238+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:22:28.196 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:27.829+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:28.196 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:27.829+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:28.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:27.829+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:28.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:27.830+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:22:28.696 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:28.319+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:28.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:28.319+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:28.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:28.319+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:28.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:28.320+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:22:28.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:28.264+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:28.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:28.264+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:28.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:28.264+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:28.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:28.265+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:22:29.196 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:28.830+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:29.196 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:28.830+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:29.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:28.830+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:29.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:28.831+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:22:29.696 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:29.278+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:29.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:29.278+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:29.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:29.278+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:29.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:29.279+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:22:29.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:29.292+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:29.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:29.292+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:29.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:29.292+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:29.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:29.293+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:22:30.196 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:29.863+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:30.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:29.863+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:30.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:29.863+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:30.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:29.864+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:22:30.696 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:30.289+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:30.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:30.289+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:30.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:30.289+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:30.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:30.290+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:22:30.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:30.253+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:30.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:30.253+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:30.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:30.253+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:30.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:30.254+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:22:31.127 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:22:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:22:31.105+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 1439 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:22:31.127 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:30.874+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:31.127 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:30.874+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:31.127 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:30.874+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:31.128 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:30.875+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:22:31.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:31.267+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:31.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:31.267+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:31.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:31.267+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:31.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:31.268+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:22:31.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:31.233+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:31.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:31.233+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:31.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:31.233+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:31.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:31.234+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:22:32.196 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:31.921+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:32.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:31.921+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:32.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:31.921+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:32.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:31.922+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:22:32.619 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:32.203+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:32.619 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:32.203+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:32.619 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:32.203+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:32.619 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:32.204+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:22:32.620 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:32.244+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:32.620 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:32.244+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:32.620 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:32.244+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:32.620 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:32.244+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:22:33.196 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:32.934+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:33.196 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:32.934+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:33.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:32.934+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:33.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:32.936+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:22:33.528 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:33.196+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:33.529 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:33.196+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:33.529 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:33.196+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:33.529 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:33.197+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:22:33.529 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:33.240+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:33.529 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:33.240+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:33.530 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:33.240+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:33.530 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:33.240+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:22:34.196 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:34.160+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:34.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:34.160+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:34.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:34.160+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:34.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:34.161+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:22:34.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:33.937+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:34.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:33.937+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:34.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:33.937+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:34.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:33.938+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:22:34.696 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:34.248+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:34.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:34.248+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:34.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:34.248+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:34.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:34.248+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:22:35.283 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:35.120+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:35.284 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:35.120+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:35.284 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:35.120+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:35.284 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:35.120+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:22:35.284 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:34.966+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:35.284 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:34.966+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:35.285 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:34.966+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:35.285 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:34.967+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:22:35.696 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:35.282+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:35.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:35.282+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:35.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:35.282+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:35.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:35.284+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:22:36.196 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:22:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:22:36.105+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 1537 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:22:36.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:35.943+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:36.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:35.943+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:36.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:35.943+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:36.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:35.944+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:22:36.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:36.137+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:36.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:36.137+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:36.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:36.137+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:36.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:36.138+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:22:36.696 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:36.247+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:36.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:36.247+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:36.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:36.247+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:36.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:36.248+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:22:37.196 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:37.103+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:37.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:37.103+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:37.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:37.103+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:37.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:37.104+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:22:37.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:36.904+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:37.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:36.904+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:37.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:36.904+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:37.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:36.905+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:22:37.671 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:37.247+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:37.673 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:37.247+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:37.673 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:37.247+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:37.673 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:37.248+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:22:37.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:37.906+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:37.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:37.906+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:37.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:37.906+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:37.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:37.907+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:22:38.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:38.142+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:38.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:38.142+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:38.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:38.142+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:38.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:38.142+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:22:38.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:38.213+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:38.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:38.213+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:38.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:38.213+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:38.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:38.214+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:22:39.168 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:38.909+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:39.169 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:38.909+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:39.169 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:38.909+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:39.169 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:38.909+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:22:39.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:39.190+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:39.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:39.190+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:39.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:39.190+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:39.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:39.190+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:22:39.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:39.168+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:39.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:39.168+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:39.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:39.168+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:39.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:39.168+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:22:40.151 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:39.897+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:40.152 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:39.897+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:40.152 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:39.897+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:40.152 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:39.898+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:22:40.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:40.178+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:40.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:40.178+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:40.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:40.178+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:40.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:40.179+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:22:40.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:40.150+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:40.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:40.150+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:40.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:40.150+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:40.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:40.151+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:22:41.196 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:22:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:22:41.106+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 1632 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:22:41.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:40.919+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:41.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:40.919+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:41.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:40.919+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:41.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:40.920+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:22:41.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:41.111+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:41.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:41.111+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:41.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:41.111+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:41.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:41.113+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:22:41.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:41.210+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:41.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:41.210+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:41.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:41.210+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:41.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:41.211+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:22:42.196 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:42.162+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:42.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:42.162+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:42.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:42.162+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:42.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:42.163+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:22:42.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:42.103+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:42.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:42.103+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:42.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:42.103+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:42.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:42.104+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:22:42.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:41.937+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:42.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:41.937+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:42.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:41.937+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:42.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:41.938+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:22:43.115 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:42.921+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:43.115 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:42.921+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:43.116 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:42.921+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:43.116 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:42.922+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:22:43.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:43.114+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:43.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:43.114+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:43.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:43.114+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:43.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:43.115+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:22:43.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:43.163+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:43.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:43.163+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:43.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:43.163+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:43.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:43.164+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:22:44.196 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:43.901+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:44.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:43.901+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:44.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:43.901+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:44.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:43.901+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:22:44.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:44.141+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:44.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:44.141+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:44.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:44.141+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:44.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:44.142+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:22:44.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:44.147+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:44.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:44.147+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:44.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:44.147+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:44.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:44.148+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:22:45.171 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:44.899+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:45.172 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:44.899+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:45.172 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:44.899+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:45.172 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:44.900+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:22:45.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:45.171+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:45.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:45.171+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:45.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:45.171+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:45.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:45.171+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:22:45.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:45.191+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:45.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:45.191+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:45.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:45.191+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:45.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:45.192+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:22:46.130 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:45.869+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:46.130 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:45.869+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:46.131 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:45.869+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:46.131 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:45.870+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:22:46.131 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:22:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:22:46.107+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 1730 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:22:46.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:46.142+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:46.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:46.142+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:46.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:46.142+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:46.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:46.142+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:22:46.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:46.235+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:46.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:46.235+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:46.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:46.235+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:46.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:46.236+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:22:47.196 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:46.908+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:47.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:46.908+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:47.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:46.908+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:47.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:46.909+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:22:47.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:47.101+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:47.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:47.101+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:47.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:47.101+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:47.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:47.102+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:22:47.696 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:47.268+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:47.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:47.268+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:47.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:47.268+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:47.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:47.269+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:22:48.067 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:47.921+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:48.067 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:47.921+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:48.067 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:47.921+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:48.067 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:47.922+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:22:48.319 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:48.066+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:48.319 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:48.066+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:48.320 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:48.066+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:48.320 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:48.066+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:22:48.696 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:48.318+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:48.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:48.318+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:48.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:48.318+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:48.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:48.319+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:22:49.196 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:49.098+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:49.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:49.098+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:49.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:49.098+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:49.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:49.099+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:22:49.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:48.880+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:49.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:48.880+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:49.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:48.880+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:49.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:48.881+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:22:49.696 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:49.292+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:49.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:49.292+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:49.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:49.292+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:49.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:49.293+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:22:50.196 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:50.075+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:50.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:50.075+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:50.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:50.075+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:50.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:50.076+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:22:50.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:49.860+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:50.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:49.860+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:50.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:49.860+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:50.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:49.861+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:22:50.696 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:50.288+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:50.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:50.288+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:50.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:50.288+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:50.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:50.289+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:22:51.109 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:50.840+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:51.109 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:50.840+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:51.109 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:50.840+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:51.109 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:50.841+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:22:51.446 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:22:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:22:51.107+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 1825 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:22:51.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:51.109+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:51.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:51.109+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:51.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:51.109+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:51.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:51.111+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:22:51.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:51.247+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:51.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:51.247+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:51.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:51.247+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:51.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:51.247+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:22:52.196 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:52.106+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:52.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:52.106+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:52.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:52.106+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:52.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:52.106+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:22:52.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:51.858+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:52.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:51.858+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:52.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:51.858+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:52.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:51.859+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:22:52.696 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:52.244+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:52.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:52.244+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:52.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:52.244+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:52.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:52.244+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:22:53.138 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:52.854+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:53.138 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:52.854+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:53.138 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:52.854+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:53.139 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:52.855+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:22:53.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:53.137+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:53.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:53.137+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:53.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:53.137+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:53.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:53.138+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:22:53.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:53.233+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:53.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:53.233+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:53.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:53.233+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:53.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:53.234+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:22:54.133 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:53.835+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:54.134 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:53.835+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:54.134 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:53.835+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:54.134 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:53.836+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:22:54.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:54.133+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:54.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:54.133+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:54.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:54.133+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:54.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:54.134+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:22:54.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:54.204+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:54.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:54.204+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:54.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:54.204+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:54.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:54.204+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:22:55.159 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:54.805+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:55.160 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:54.805+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:55.160 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:54.805+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:55.160 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:54.806+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:22:55.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:55.162+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:55.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:55.162+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:55.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:55.162+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:55.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:55.163+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:22:55.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:55.158+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:55.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:55.158+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:55.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:55.158+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:55.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:55.159+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:22:56.109 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:55.771+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:56.109 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:55.771+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:56.109 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:55.771+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:56.109 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:55.772+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:22:56.446 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:22:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:22:56.108+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 1923 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:22:56.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:56.134+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:56.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:56.134+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:56.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:56.134+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:56.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:56.134+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:22:56.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:56.139+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:56.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:56.139+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:56.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:56.139+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:56.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:56.140+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:22:57.115 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:56.781+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:57.116 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:56.781+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:57.116 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:56.781+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:57.116 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:56.782+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:22:57.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:57.119+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:57.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:57.119+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:57.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:57.119+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:57.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:57.120+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:22:57.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:57.114+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:57.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:57.114+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:57.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:57.114+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:57.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:57.116+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:22:58.089 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:57.741+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:58.089 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:57.741+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:58.089 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:57.741+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:58.089 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:57.741+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:22:58.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:58.126+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:58.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:58.126+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:58.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:58.126+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:58.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:58.127+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:22:58.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:58.089+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:58.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:58.089+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:58.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:58.089+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:58.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:58.089+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:22:59.078 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:58.765+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:59.078 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:58.765+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:59.079 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:58.765+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:22:59.079 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:58.766+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:22:59.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:59.147+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:59.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:59.147+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:59.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:59.147+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:22:59.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:22:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:22:59.147+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:22:59.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:59.078+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:59.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:59.078+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:59.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:59.078+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:22:59.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:22:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:22:59.078+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:23:00.106 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:59.801+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:00.107 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:59.801+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:00.107 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:59.801+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:00.107 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:22:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:22:59.802+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:23:00.107 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:00.040+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:00.108 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:00.040+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:00.108 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:00.040+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:00.108 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:00.041+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:23:00.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:00.107+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:00.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:00.107+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:00.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:00.107+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:00.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:00.107+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:23:01.110 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:00.800+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:01.110 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:00.800+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:01.110 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:00.800+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:01.110 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:00.800+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:23:01.111 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:01.019+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:01.111 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:01.019+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:01.111 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:01.019+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:01.111 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:01.020+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:23:01.446 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:23:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:23:01.110+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 2018 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:23:01.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:01.154+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:01.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:01.154+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:01.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:01.154+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:01.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:01.155+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:23:02.112 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:02.031+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:02.112 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:02.031+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:02.113 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:02.031+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:02.113 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:02.032+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:23:02.113 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:01.847+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:02.113 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:01.847+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:02.113 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:01.847+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:02.114 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:01.848+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:23:02.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:02.112+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:02.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:02.112+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:02.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:02.112+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:02.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:02.113+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:23:03.127 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:03.078+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:03.127 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:03.078+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:03.127 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:03.078+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:03.127 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:03.079+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:23:03.128 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:02.873+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:03.128 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:02.873+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:03.128 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:02.873+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:03.128 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:02.874+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:23:03.129 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:03.127+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:03.129 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:03.127+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:03.129 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:03.127+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:03.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:03.128+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:23:04.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:03.917+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:04.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:03.917+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:04.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:03.917+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:04.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:03.918+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:23:04.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:04.112+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:04.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:04.112+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:04.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:04.112+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:04.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:04.113+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:23:04.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:04.068+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:04.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:04.068+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:04.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:04.068+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:04.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:04.069+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:23:05.196 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:04.925+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:05.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:04.925+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:05.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:04.925+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:05.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:04.926+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:23:05.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:05.113+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:05.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:05.113+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:05.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:05.113+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:05.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:05.113+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:23:05.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:05.096+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:05.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:05.096+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:05.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:05.096+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:05.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:05.097+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:23:06.196 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:23:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:23:06.109+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 2116 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:23:06.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:05.896+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:06.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:05.896+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:06.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:05.896+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:06.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:05.897+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:23:06.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:06.084+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:06.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:06.084+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:06.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:06.084+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:06.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:06.085+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:23:06.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:06.077+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:06.199 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:06.077+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:06.199 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:06.077+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:06.199 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:06.077+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:23:07.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:06.874+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:07.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:06.874+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:07.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:06.874+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:07.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:06.875+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:23:07.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:07.087+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:07.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:07.087+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:07.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:07.087+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:07.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:07.088+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:23:07.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:07.053+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:07.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:07.053+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:07.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:07.053+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:07.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:07.054+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:23:08.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:08.132+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:08.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:08.132+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:08.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:08.132+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:08.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:08.132+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:23:08.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:07.919+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:08.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:07.919+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:08.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:07.919+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:08.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:07.920+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:23:08.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:08.071+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:08.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:08.071+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:08.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:08.071+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:08.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:08.072+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:23:09.196 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:08.945+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:09.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:08.945+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:09.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:08.945+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:09.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:08.946+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:23:09.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:09.178+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:09.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:09.178+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:09.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:09.178+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:09.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:09.179+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:23:09.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:09.037+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:09.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:09.037+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:09.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:09.037+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:09.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:09.038+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:23:09.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:09.938+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:09.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:09.938+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:09.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:09.938+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:09.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:09.939+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:23:10.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:10.184+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:10.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:10.184+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:10.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:10.184+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:10.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:10.184+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:23:10.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:10.072+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:10.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:10.072+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:10.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:10.072+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:10.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:10.073+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:23:11.190 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:10.891+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:11.190 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:10.891+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:11.190 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:10.891+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:11.190 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:10.892+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:23:11.191 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:11.030+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:11.191 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:11.030+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:11.191 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:11.030+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:11.191 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:11.030+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:23:11.191 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:23:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:23:11.110+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 2211 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:23:11.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:11.189+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:11.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:11.189+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:11.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:11.189+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:11.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:11.190+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:23:12.172 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:11.872+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:12.172 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:11.872+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:12.173 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:11.872+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:12.173 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:11.873+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:23:12.173 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:12.013+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:12.173 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:12.013+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:12.173 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:12.013+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:12.173 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:12.014+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:23:12.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:12.171+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:12.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:12.171+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:12.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:12.171+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:12.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:12.172+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:23:13.030 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:12.871+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:13.031 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:12.871+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:13.031 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:12.871+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:13.031 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:12.872+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:23:13.031 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:12.975+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:13.031 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:12.975+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:13.032 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:12.975+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:13.032 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:12.976+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:23:13.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:13.214+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:13.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:13.214+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:13.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:13.214+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:13.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:13.215+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:23:14.196 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:13.870+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:14.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:13.870+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:14.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:13.870+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:14.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:13.870+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:23:14.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:13.929+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:14.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:13.929+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:14.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:13.929+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:14.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:13.930+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:23:14.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:14.240+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:14.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:14.240+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:14.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:14.240+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:14.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:14.241+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:23:15.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:14.835+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:15.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:14.835+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:15.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:14.835+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:15.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:14.836+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:23:15.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:14.917+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:15.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:14.917+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:15.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:14.917+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:15.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:14.918+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:23:15.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:15.258+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:15.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:15.258+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:15.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:15.258+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:15.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:15.259+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:23:16.111 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:15.899+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:16.112 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:15.899+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:16.112 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:15.899+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:16.112 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:15.900+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:23:16.112 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:15.850+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:16.112 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:15.850+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:16.112 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:15.850+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:16.113 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:15.851+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:23:16.446 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:23:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:23:16.110+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 2309 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:23:16.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:16.251+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:16.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:16.251+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:16.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:16.251+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:16.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:16.252+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:23:17.196 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:16.885+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:17.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:16.885+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:17.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:16.885+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:17.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:16.886+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:23:17.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:16.930+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:17.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:16.931+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:17.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:16.931+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:17.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:16.931+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:23:17.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:17.254+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:17.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:17.254+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:17.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:17.254+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:17.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:17.254+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:23:18.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:17.865+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:18.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:17.865+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:18.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:17.865+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:18.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:17.865+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:23:18.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:17.889+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:18.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:17.889+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:18.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:17.889+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:18.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:17.890+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:23:18.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:18.279+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:18.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:18.279+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:18.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:18.279+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:18.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:18.279+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:23:19.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:18.905+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:19.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:18.905+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:19.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:18.905+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:19.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:18.906+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:23:19.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:18.884+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:19.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:18.884+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:19.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:18.884+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:19.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:18.884+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:23:19.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:19.264+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:19.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:19.264+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:19.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:19.264+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:19.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:19.265+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:23:20.196 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:19.883+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:20.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:19.883+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:20.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:19.883+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:20.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:19.884+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:23:20.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:19.927+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:20.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:19.927+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:20.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:19.927+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:20.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:19.928+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:23:20.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:20.234+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:20.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:20.234+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:20.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:20.234+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:20.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:20.235+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:23:21.112 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:20.933+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:21.112 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:20.933+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:21.113 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:20.933+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:21.113 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:20.934+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:23:21.113 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:20.856+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:21.113 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:20.856+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:21.113 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:20.856+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:21.113 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:20.857+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:23:21.446 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:23:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:23:21.111+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 2404 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:23:21.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:21.269+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:21.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:21.269+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:21.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:21.269+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:21.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:21.270+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:23:22.196 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:21.936+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:22.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:21.936+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:22.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:21.936+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:22.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:21.937+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:23:22.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:21.854+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:22.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:21.854+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:22.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:21.854+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:22.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:21.855+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:23:22.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:22.275+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:22.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:22.275+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:22.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:22.275+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:22.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:22.276+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:23:23.131 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:22.944+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:23.131 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:22.944+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:23.131 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:22.944+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:23.131 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:22.945+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:23:23.131 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:22.847+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:23.132 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:22.847+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:23.132 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:22.847+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:23.132 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:22.848+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:23:23.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:23.308+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:23.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:23.308+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:23.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:23.308+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:23.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:23.308+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:23:24.196 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:23.865+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:24.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:23.865+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:24.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:23.865+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:24.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:23.866+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:23:24.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:23.913+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:24.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:23.913+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:24.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:23.913+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:24.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:23.914+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:23:24.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:24.302+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:24.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:24.302+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:24.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:24.302+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:24.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:24.302+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:23:25.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:24.823+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:25.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:24.823+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:25.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:24.823+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:25.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:24.824+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:23:25.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:24.964+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:25.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:24.964+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:25.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:24.964+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:25.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:24.965+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:23:25.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:25.264+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:25.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:25.264+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:25.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:25.264+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:25.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:25.265+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:23:26.113 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:25.931+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:26.113 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:25.931+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:26.113 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:25.931+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:26.113 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:25.932+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:23:26.113 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:25.795+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:26.114 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:25.795+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:26.114 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:25.795+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:26.114 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:25.796+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:23:26.446 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:23:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:23:26.112+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 2502 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:23:26.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:26.308+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:26.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:26.308+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:26.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:26.308+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:26.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:26.309+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:23:27.196 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:26.830+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:27.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:26.830+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:27.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:26.830+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:27.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:26.831+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:23:27.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:26.917+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:27.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:26.917+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:27.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:26.917+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:27.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:26.918+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:23:27.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:27.342+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:27.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:27.342+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:27.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:27.342+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:27.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:27.342+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:23:28.187 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:27.921+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:28.187 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:27.921+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:28.187 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:27.921+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:28.187 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:27.922+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:23:28.188 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:27.838+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:28.188 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:27.838+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:28.188 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:27.838+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:28.188 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:27.839+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:23:28.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:28.381+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:28.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:28.381+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:28.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:28.381+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:28.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:28.382+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:23:29.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:28.827+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:29.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:28.827+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:29.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:28.827+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:29.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:28.828+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:23:29.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:28.949+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:29.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:28.949+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:29.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:28.949+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:29.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:28.950+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:23:29.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:29.383+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:29.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:29.383+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:29.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:29.383+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:29.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:29.384+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:23:30.196 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:29.861+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:30.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:29.861+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:30.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:29.861+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:30.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:29.862+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:23:30.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:29.904+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:30.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:29.904+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:30.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:29.904+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:30.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:29.905+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:23:30.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:30.395+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:30.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:30.395+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:30.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:30.395+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:30.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:30.396+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:23:31.113 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:30.915+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:31.114 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:30.915+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:31.114 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:30.915+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:31.114 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:30.916+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:23:31.114 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:30.858+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:31.114 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:30.858+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:31.115 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:30.858+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:31.115 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:30.859+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:23:31.409 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:23:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:23:31.112+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 2597 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:23:31.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:31.408+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:31.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:31.408+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:31.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:31.408+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:31.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:31.410+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:23:32.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:31.860+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:32.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:31.860+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:32.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:31.860+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:32.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:31.861+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:23:32.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:31.897+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:32.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:31.897+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:32.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:31.897+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:32.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:31.899+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:23:32.838 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:32.449+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:32.838 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:32.449+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:32.838 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:32.449+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:32.838 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:32.449+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:23:33.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:32.890+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:33.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:32.890+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:33.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:32.890+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:33.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:32.891+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:23:33.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:32.837+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:33.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:32.837+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:33.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:32.837+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:33.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:32.837+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:23:33.809 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:33.489+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:33.809 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:33.489+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:33.809 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:33.489+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:33.809 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:33.490+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:23:34.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:33.808+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:34.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:33.808+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:34.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:33.808+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:34.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:33.809+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:23:34.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:33.934+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:34.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:33.934+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:34.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:33.934+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:34.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:33.935+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:23:34.787 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:34.518+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:34.788 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:34.518+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:34.788 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:34.518+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:34.788 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:34.519+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:23:35.196 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:34.786+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:35.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:34.786+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:35.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:34.786+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:35.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:34.787+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:23:35.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:34.978+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:35.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:34.978+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:35.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:34.978+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:35.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:34.979+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:23:35.946 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:35.565+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:35.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:35.565+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:35.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:35.565+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:35.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:35.565+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:23:35.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:35.781+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:35.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:35.781+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:35.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:35.781+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:35.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:35.782+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:23:36.235 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:23:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:23:36.113+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 2695 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:23:36.235 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:35.950+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:36.235 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:35.950+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:36.236 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:35.950+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:36.236 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:35.950+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:23:36.786 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:36.532+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:36.786 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:36.532+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:36.786 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:36.532+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:36.787 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:36.533+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:23:37.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:36.916+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:37.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:36.916+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:37.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:36.916+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:37.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:36.917+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:23:37.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:36.785+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:37.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:36.785+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:37.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:36.785+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:37.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:36.786+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:23:37.881 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:37.530+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:37.881 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:37.530+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:37.881 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:37.530+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:37.881 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:37.531+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:23:37.882 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:37.736+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:37.882 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:37.736+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:37.882 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:37.736+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:37.882 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:37.737+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:23:38.196 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:37.879+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:38.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:37.879+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:38.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:37.879+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:38.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:37.881+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:23:38.848 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:38.552+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:38.848 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:38.552+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:38.848 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:38.552+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:38.848 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:38.553+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:23:38.849 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:38.691+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:38.849 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:38.691+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:38.849 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:38.691+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:38.849 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:38.692+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:23:39.196 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:38.847+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:39.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:38.847+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:39.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:38.847+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:39.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:38.848+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:23:39.832 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:39.695+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:39.832 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:39.695+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:39.832 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:39.695+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:39.832 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:39.696+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:23:39.833 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:39.506+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:39.833 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:39.506+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:39.833 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:39.506+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:39.833 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:39.507+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:23:40.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:39.893+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:40.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:39.893+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:40.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:39.893+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:40.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:39.893+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:23:40.853 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:40.521+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:40.854 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:40.521+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:40.854 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:40.521+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:40.854 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:40.522+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:23:40.854 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:40.656+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:40.855 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:40.656+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:40.855 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:40.656+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:40.855 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:40.656+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:23:41.115 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:40.852+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:41.115 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:40.852+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:41.115 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:40.852+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:41.115 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:40.854+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:23:41.446 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:23:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:23:41.114+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 2790 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:23:41.829 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:41.625+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:41.829 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:41.625+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:41.829 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:41.625+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:41.829 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:41.626+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:23:41.829 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:41.492+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:41.830 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:41.492+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:41.830 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:41.492+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:41.830 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:41.493+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:23:42.196 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:41.827+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:42.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:41.827+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:42.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:41.827+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:42.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:41.829+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:23:42.783 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:42.453+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:42.784 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:42.453+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:42.784 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:42.453+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:42.784 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:42.455+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:23:42.784 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:42.635+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:42.784 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:42.635+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:42.785 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:42.635+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:42.785 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:42.635+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:23:43.196 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:42.783+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:43.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:42.783+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:43.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:42.783+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:43.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:42.783+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:23:43.609 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:43.433+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:43.609 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:43.433+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:43.609 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:43.433+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:43.609 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:43.434+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:23:43.609 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:43.608+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:43.610 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:43.608+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:43.610 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:43.608+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:43.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:43.609+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:23:43.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:43.825+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:43.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:43.825+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:43.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:43.825+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:43.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:43.825+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:23:44.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:44.437+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:44.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:44.437+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:44.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:44.437+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:44.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:44.439+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:23:44.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:44.565+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:44.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:44.565+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:44.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:44.565+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:44.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:44.565+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:23:45.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:44.846+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:45.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:44.846+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:45.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:44.846+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:45.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:44.846+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:23:45.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:45.390+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:45.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:45.390+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:45.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:45.390+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:45.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:45.391+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:23:45.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:45.556+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:45.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:45.556+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:45.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:45.556+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:45.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:45.557+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:23:46.115 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:45.811+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:46.116 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:45.811+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:46.116 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:45.811+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:46.116 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:45.812+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:23:46.405 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:23:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:23:46.115+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 2886 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:23:46.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:46.405+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:46.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:46.405+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:46.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:46.405+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:46.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:46.406+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:23:46.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:46.606+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:46.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:46.606+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:46.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:46.606+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:46.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:46.607+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:23:47.196 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:46.832+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:47.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:46.832+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:47.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:46.832+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:47.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:46.833+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:23:47.633 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:47.382+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:47.634 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:47.382+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:47.634 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:47.382+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:47.634 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:47.383+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:23:47.946 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:47.800+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:47.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:47.800+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:47.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:47.800+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:47.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:47.801+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:23:47.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:47.633+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:47.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:47.633+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:47.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:47.633+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:47.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:47.634+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:23:48.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:48.367+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:48.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:48.367+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:48.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:48.367+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:48.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:48.368+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:23:48.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:48.590+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:48.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:48.590+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:48.699 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:48.590+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:48.699 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:48.591+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:23:49.196 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:48.768+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:49.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:48.768+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:49.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:48.768+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:49.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:48.769+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:23:49.625 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:49.334+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:49.625 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:49.334+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:49.625 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:49.334+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:49.626 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:49.335+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:23:49.946 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:49.800+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:49.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:49.800+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:49.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:49.800+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:49.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:49.801+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:23:49.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:49.624+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:49.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:49.624+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:49.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:49.624+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:49.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:49.625+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:23:50.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:50.370+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:50.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:50.370+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:50.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:50.370+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:50.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:50.371+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:23:50.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:50.608+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:50.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:50.608+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:50.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:50.608+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:50.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:50.609+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:23:51.116 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:50.763+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:51.117 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:50.763+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:51.117 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:50.763+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:51.117 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:50.764+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:23:51.446 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:23:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:23:51.115+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 2980 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:23:51.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:51.339+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:51.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:51.339+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:51.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:51.339+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:51.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:51.340+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:23:51.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:51.575+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:51.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:51.575+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:51.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:51.575+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:51.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:51.576+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:23:51.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:51.784+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:51.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:51.784+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:51.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:51.784+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:51.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:51.785+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:23:52.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:52.379+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:52.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:52.379+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:52.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:52.379+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:52.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:52.380+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:23:52.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:52.603+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:52.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:52.603+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:52.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:52.603+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:52.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:52.604+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:23:53.196 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:52.742+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:53.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:52.742+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:53.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:52.742+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:53.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:52.743+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:23:53.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:53.369+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:53.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:53.369+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:53.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:53.369+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:53.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:53.370+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:23:53.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:53.559+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:53.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:53.559+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:53.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:53.559+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:53.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:53.560+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:23:54.196 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:53.771+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:54.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:53.771+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:54.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:53.771+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:54.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:53.772+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:23:54.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:54.387+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:54.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:54.387+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:54.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:54.387+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:54.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:54.388+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:23:54.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:54.595+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:54.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:54.595+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:54.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:54.595+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:54.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:54.596+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:23:55.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:54.775+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:55.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:54.775+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:55.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:54.775+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:55.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:54.776+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:23:55.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:55.401+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:55.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:55.401+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:55.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:55.401+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:55.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:55.402+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:23:55.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:55.613+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:55.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:55.613+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:55.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:55.613+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:55.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:55.614+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:23:56.117 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:55.729+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:56.117 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:55.729+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:56.117 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:55.729+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:56.118 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:55.730+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:23:56.412 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:23:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:23:56.116+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 3077 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:23:56.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:56.411+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:56.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:56.411+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:56.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:56.411+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:56.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:56.412+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:23:56.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:56.598+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:56.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:56.598+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:56.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:56.598+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:56.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:56.599+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:23:57.196 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:56.708+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:57.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:56.708+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:57.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:56.708+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:57.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:56.709+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:23:57.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:57.558+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:57.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:57.558+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:57.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:57.558+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:57.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:57.559+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:23:57.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:57.395+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:57.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:57.395+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:57.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:57.395+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:57.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:57.395+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:23:58.196 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:57.702+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:58.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:57.702+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:58.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:57.702+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:58.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:57.703+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:23:58.659 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:58.355+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:58.659 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:58.355+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:58.659 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:58.355+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:58.659 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:58.356+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:23:58.660 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:58.540+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:58.660 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:58.540+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:58.660 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:58.540+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:58.660 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:58.541+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:23:58.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:58.656+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:58.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:58.656+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:58.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:58.656+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:58.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:58.657+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:23:59.623 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:59.549+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:59.623 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:59.549+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:59.623 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:59.549+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:23:59.623 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:23:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:23:59.550+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:23:59.624 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:59.312+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:59.624 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:59.312+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:59.624 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:59.312+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:23:59.624 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:23:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:23:59.313+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:23:59.946 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:59.622+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:59.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:59.622+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:59.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:59.622+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:23:59.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:23:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:23:59.623+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:24:00.558 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:00.290+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:00.558 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:00.290+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:00.558 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:00.290+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:00.558 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:00.291+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:24:00.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:00.557+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:00.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:00.557+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:00.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:00.557+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:00.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:00.558+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:24:00.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:00.669+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:00.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:00.669+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:00.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:00.669+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:00.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:00.669+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:24:01.447 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:24:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:24:01.117+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 3172 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:24:01.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:01.266+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:01.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:01.266+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:01.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:01.266+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:01.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:01.267+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:24:01.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:01.516+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:01.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:01.516+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:01.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:01.516+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:01.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:01.517+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:24:01.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:01.622+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:01.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:01.622+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:01.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:01.622+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:01.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:01.623+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:24:02.645 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:02.239+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:02.645 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:02.239+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:02.645 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:02.239+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:02.645 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:02.240+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:24:02.646 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:02.468+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:02.646 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:02.468+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:02.646 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:02.468+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:02.646 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:02.469+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:24:02.646 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:02.644+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:02.646 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:02.644+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:02.647 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:02.644+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:02.946 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:02.645+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:24:03.515 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:03.206+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:03.515 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:03.206+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:03.515 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:03.206+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:03.515 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:03.207+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:24:03.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:03.630+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:03.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:03.630+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:03.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:03.630+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:03.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:03.631+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:24:03.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:03.513+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:03.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:03.513+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:03.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:03.513+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:03.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:03.515+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:24:04.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:04.179+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:04.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:04.179+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:04.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:04.179+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:04.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:04.180+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:24:04.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:04.525+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:04.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:04.525+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:04.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:04.525+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:04.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:04.526+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:24:04.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:04.587+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:04.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:04.587+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:04.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:04.587+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:04.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:04.588+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:24:05.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:05.211+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:05.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:05.211+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:05.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:05.211+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:05.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:05.211+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:24:05.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:05.539+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:05.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:05.539+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:05.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:05.539+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:05.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:05.541+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:24:05.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:05.548+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:05.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:05.548+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:05.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:05.548+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:05.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:05.549+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:24:06.446 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:24:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:24:06.117+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 3269 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:24:06.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:06.207+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:06.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:06.207+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:06.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:06.207+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:06.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:06.208+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:24:06.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:06.527+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:06.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:06.527+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:06.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:06.527+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:06.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:06.528+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:24:06.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:06.503+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:06.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:06.503+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:06.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:06.503+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:06.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:06.503+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:24:07.503 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:07.236+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:07.504 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:07.236+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:07.504 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:07.236+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:07.504 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:07.237+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:24:07.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:07.502+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:07.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:07.502+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:07.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:07.502+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:07.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:07.503+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:24:07.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:07.522+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:07.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:07.522+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:07.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:07.522+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:07.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:07.522+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:24:08.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:08.205+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:08.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:08.205+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:08.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:08.205+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:08.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:08.205+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:24:08.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:08.478+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:08.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:08.478+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:08.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:08.478+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:08.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:08.479+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:24:08.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:08.550+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:08.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:08.550+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:08.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:08.550+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:08.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:08.551+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:24:09.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:09.166+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:09.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:09.166+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:09.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:09.166+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:09.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:09.167+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:24:09.832 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:09.458+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:09.832 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:09.458+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:09.832 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:09.458+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:09.833 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:09.459+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:24:09.833 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:09.589+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:09.833 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:09.589+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:09.833 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:09.589+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:09.833 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:09.590+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:24:10.419 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:10.155+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:10.419 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:10.155+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:10.419 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:10.155+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:10.420 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:10.156+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:24:10.696 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:10.418+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:10.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:10.418+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:10.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:10.418+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:10.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:10.419+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:24:10.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:10.584+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:10.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:10.584+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:10.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:10.584+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:10.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:10.585+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:24:11.414 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:24:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:24:11.118+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 3363 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:24:11.414 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:11.154+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:11.414 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:11.154+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:11.415 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:11.154+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:11.415 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:11.155+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:24:11.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:11.413+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:11.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:11.413+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:11.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:11.413+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:11.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:11.414+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:24:11.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:11.620+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:11.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:11.620+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:11.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:11.620+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:11.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:11.621+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:24:12.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:12.399+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:12.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:12.399+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:12.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:12.399+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:12.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:12.400+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:24:12.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:12.183+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:12.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:12.183+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:12.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:12.183+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:12.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:12.183+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:24:12.946 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:12.590+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:12.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:12.590+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:12.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:12.590+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:12.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:12.591+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:24:13.433 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:13.169+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:13.433 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:13.169+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:13.433 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:13.169+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:13.433 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:13.170+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:24:13.696 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:13.432+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:13.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:13.432+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:13.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:13.432+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:13.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:13.433+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:24:13.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:13.636+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:13.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:13.636+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:13.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:13.636+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:13.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:13.637+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:24:14.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:14.131+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:14.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:14.131+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:14.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:14.131+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:14.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:14.132+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:24:14.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:14.636+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:14.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:14.636+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:14.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:14.636+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:14.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:14.636+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:24:14.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:14.459+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:14.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:14.459+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:14.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:14.459+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:14.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:14.460+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:24:15.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:15.144+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:15.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:15.144+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:15.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:15.144+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:15.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:15.145+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:24:15.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:15.499+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:15.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:15.499+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:15.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:15.499+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:15.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:15.500+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:24:15.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:15.682+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:15.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:15.682+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:15.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:15.682+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:15.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:15.683+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:24:16.447 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:24:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:24:16.119+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 3460 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:24:16.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:16.134+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:16.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:16.134+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:16.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:16.134+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:16.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:16.135+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:24:16.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:16.471+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:16.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:16.471+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:16.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:16.471+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:16.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:16.472+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:24:16.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:16.693+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:16.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:16.693+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:16.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:16.693+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:16.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:16.694+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:24:17.436 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:17.115+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:17.437 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:17.115+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:17.437 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:17.115+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:17.437 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:17.116+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:24:17.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:17.436+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:17.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:17.436+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:17.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:17.436+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:17.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:17.437+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:24:17.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:17.672+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:17.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:17.672+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:17.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:17.672+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:17.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:17.673+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:24:18.437 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:18.093+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:18.438 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:18.093+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:18.438 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:18.093+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:18.438 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:18.094+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:24:18.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:18.662+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:18.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:18.662+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:18.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:18.662+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:18.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:18.663+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:24:18.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:18.436+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:18.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:18.436+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:18.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:18.436+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:18.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:18.437+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:24:19.401 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:19.136+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:19.402 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:19.136+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:19.402 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:19.136+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:19.402 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:19.137+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:24:19.471 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-05-04T12:24:19.472 WARNING:tasks.kclient:Ordinary umount failed, forcing... 2024-05-04T12:24:19.472 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-05-04T12:24:19.472 DEBUG:teuthology.orchestra.run.smithi079:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-04T12:24:19.490 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-05-04T12:24:19.491 DEBUG:teuthology.orchestra.run.smithi079:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-04T12:24:19.547 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-04T12:24:19.681 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:19.679+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:19.682 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:19.679+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:19.682 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:19.400+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:19.682 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:19.400+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:19.682 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:19.400+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:19.682 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:19.401+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:24:19.946 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:19.679+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:19.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:19.681+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:24:20.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:20.186+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:20.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:20.186+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:20.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:20.186+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:20.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:20.187+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:24:20.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:20.417+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:20.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:20.417+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:20.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:20.418+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:20.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:20.419+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:24:20.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:20.647+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:20.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:20.647+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:20.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:20.647+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:20.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:20.648+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:24:21.402 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:24:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:24:21.120+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 3554 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:24:21.403 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:21.177+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:21.403 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:21.177+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:21.403 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:21.177+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:21.403 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:21.178+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:24:21.668 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:21.401+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:21.668 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:21.401+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:21.669 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:21.401+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:21.669 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:21.402+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:24:21.946 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:21.667+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:21.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:21.667+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:21.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:21.667+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:21.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:21.668+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:24:22.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:22.131+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:22.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:22.131+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:22.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:22.131+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:22.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:22.131+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:24:22.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:22.445+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:22.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:22.445+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:22.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:22.445+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:22.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:22.446+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:24:23.145 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:22.697+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:23.146 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:22.697+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:23.146 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:22.697+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:23.146 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:22.697+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:24:23.439 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:23.144+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:23.439 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:23.144+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:23.440 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:23.144+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:23.440 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:23.145+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:24:23.696 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:23.438+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:23.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:23.438+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:23.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:23.438+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:23.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:23.440+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:24:24.103 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:23.711+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:24.103 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:23.711+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:24.104 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:23.711+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:24.104 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:23.712+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:24:24.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:24.102+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:24.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:24.102+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:24.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:24.102+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:24.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:24.103+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:24:24.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:24.481+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:24.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:24.481+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:24.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:24.481+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:24.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:24.482+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:24:24.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:24.684+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:24.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:24.684+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:24.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:24.684+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:24.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:24.684+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:24:25.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:25.082+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:25.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:25.082+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:25.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:25.082+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:25.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:25.083+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:24:25.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:25.450+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:25.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:25.450+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:25.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:25.450+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:25.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:25.451+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:24:25.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:25.660+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:25.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:25.660+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:25.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:25.660+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:25.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:25.661+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:24:26.446 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:24:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:24:26.121+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 3652 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:24:26.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:26.095+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:26.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:26.095+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:26.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:26.095+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:26.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:26.096+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:24:26.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:26.447+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:26.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:26.447+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:26.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:26.447+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:26.699 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:26.448+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:24:27.105 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:26.697+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:27.105 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:26.697+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:27.105 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:26.697+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:27.106 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:26.698+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:24:27.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:27.104+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:27.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:27.104+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:27.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:27.104+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:27.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:27.105+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:24:27.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:27.476+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:27.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:27.476+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:27.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:27.476+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:27.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:27.477+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:24:27.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:27.679+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:27.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:27.679+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:27.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:27.679+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:27.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:27.679+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:24:28.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:28.091+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:28.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:28.091+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:28.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:28.091+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:28.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:28.092+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:24:28.800 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:28.715+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:28.800 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:28.715+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:28.800 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:28.715+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:28.800 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:28.716+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:24:28.800 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:28.473+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:28.801 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:28.473+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:28.801 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:28.473+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:28.801 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:28.474+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:24:29.432 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:29.086+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:29.432 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:29.086+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:29.432 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:29.086+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:29.433 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:29.087+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:24:29.696 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:29.431+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:29.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:29.431+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:29.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:29.431+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:29.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:29.432+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:24:30.078 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:29.701+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:30.078 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:29.701+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:30.078 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:29.701+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:30.079 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:29.702+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:24:30.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:30.077+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:30.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:30.077+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:30.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:30.077+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:30.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:30.078+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:24:30.946 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:30.692+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:30.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:30.693+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:30.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:30.693+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:30.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:30.693+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:24:30.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:30.451+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:30.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:30.451+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:30.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:30.451+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:30.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:30.452+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:24:31.446 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:24:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:24:31.122+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 3747 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:24:31.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:31.110+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:31.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:31.110+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:31.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:31.110+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:31.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:31.110+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:24:31.946 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:31.676+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:31.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:31.676+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:31.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:31.676+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:31.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:31.677+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:24:31.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:31.461+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:31.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:31.461+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:31.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:31.461+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:31.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:31.462+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:24:32.434 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:32.122+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:32.434 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:32.122+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:32.435 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:32.122+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:32.435 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:32.123+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:24:32.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:32.434+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:32.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:32.434+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:32.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:32.434+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:32.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:32.435+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:24:33.153 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:32.715+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:33.153 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:32.715+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:33.153 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:32.715+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:33.153 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:32.716+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:24:33.430 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:33.153+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:33.431 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:33.153+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:33.431 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:33.153+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:33.431 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:33.154+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:24:33.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:33.430+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:33.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:33.430+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:33.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:33.430+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:33.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:33.431+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:24:34.192 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:33.747+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:34.192 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:33.747+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:34.192 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:33.747+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:34.192 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:33.748+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:24:34.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:34.191+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:34.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:34.191+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:34.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:34.191+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:34.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:34.193+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:24:34.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:34.433+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:34.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:34.433+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:34.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:34.433+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:34.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:34.433+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:24:35.196 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:34.743+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:35.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:34.743+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:35.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:34.743+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:35.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:34.744+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:24:35.476 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:35.209+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:35.476 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:35.209+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:35.476 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:35.209+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:35.476 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:35.210+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:24:35.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:35.475+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:35.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:35.475+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:35.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:35.475+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:35.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:35.476+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:24:35.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:35.695+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:35.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:35.695+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:35.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:35.695+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:35.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:35.696+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:24:36.434 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:24:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:24:36.122+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 3845 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:24:36.435 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:36.195+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:36.435 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:36.195+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:36.435 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:36.195+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:36.435 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:36.196+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:24:36.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:36.434+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:36.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:36.434+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:36.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:36.434+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:36.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:36.435+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:24:36.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:36.666+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:36.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:36.666+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:36.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:36.666+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:36.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:36.667+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:24:37.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:37.157+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:37.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:37.157+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:37.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:37.157+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:37.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:37.158+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:24:37.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:37.483+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:37.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:37.483+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:37.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:37.483+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:37.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:37.484+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:24:37.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:37.693+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:37.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:37.693+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:37.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:37.693+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:37.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:37.694+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:24:38.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:38.140+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:38.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:38.140+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:38.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:38.140+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:38.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:38.141+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:24:38.779 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:38.685+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:38.779 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:38.685+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:38.779 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:38.685+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:38.779 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:38.686+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:24:38.779 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:38.477+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:38.780 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:38.477+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:38.780 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:38.477+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:38.780 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:38.478+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:24:39.450 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:39.188+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:39.450 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:39.188+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:39.450 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:39.188+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:39.450 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:39.189+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:24:39.832 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:39.496+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:39.832 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:39.496+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:39.833 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:39.496+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:39.833 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:39.497+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:24:39.833 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:39.658+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:39.833 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:39.658+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:39.833 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:39.658+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:39.833 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:39.659+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:24:40.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:40.160+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:40.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:40.160+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:40.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:40.160+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:40.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:40.161+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:24:40.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:40.545+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:40.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:40.545+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:40.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:40.545+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:40.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:40.546+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:24:40.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:40.679+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:40.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:40.679+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:40.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:40.679+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:40.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:40.680+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:24:41.447 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:24:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:24:41.123+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 3940 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:24:41.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:41.193+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:41.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:41.193+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:41.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:41.193+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:41.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:41.194+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:24:41.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:41.571+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:41.699 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:41.571+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:41.699 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:41.571+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:41.699 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:41.572+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:24:42.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:41.697+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:42.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:41.697+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:42.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:41.697+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:42.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:41.698+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:24:42.576 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:42.206+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:42.577 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:42.206+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:42.577 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:42.206+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:42.577 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:42.207+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:24:42.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:42.575+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:42.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:42.575+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:42.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:42.575+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:42.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:42.576+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:24:42.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:42.689+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:42.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:42.689+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:42.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:42.689+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:42.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:42.690+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:24:43.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:43.194+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:43.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:43.194+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:43.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:43.194+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:43.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:43.195+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:24:43.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:43.588+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:43.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:43.588+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:43.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:43.588+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:43.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:43.589+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:24:43.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:43.700+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:43.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:43.700+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:43.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:43.700+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:43.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:43.701+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:24:44.575 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:44.238+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:44.576 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:44.238+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:44.576 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:44.238+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:44.576 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:44.238+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:24:44.923 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:44.741+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:44.923 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:44.741+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:44.923 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:44.741+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:44.923 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:44.742+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:24:44.924 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:44.574+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:44.924 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:44.574+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:44.924 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:44.574+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:44.924 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:44.575+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:24:45.608 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:45.236+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:45.609 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:45.236+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:45.609 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:45.236+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:45.609 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:45.236+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:24:45.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:45.785+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:45.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:45.785+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:45.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:45.785+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:45.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:45.786+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:24:45.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:45.607+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:45.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:45.607+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:45.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:45.607+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:45.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:45.608+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:24:46.446 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:24:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:24:46.124+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 4038 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:24:46.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:46.256+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:46.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:46.256+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:46.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:46.256+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:46.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:46.257+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:24:46.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:46.569+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:46.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:46.569+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:46.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:46.569+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:46.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:46.569+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:24:46.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:46.749+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:46.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:46.749+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:46.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:46.749+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:46.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:46.750+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:24:47.584 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:47.296+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:47.585 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:47.296+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:47.585 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:47.296+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:47.585 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:47.296+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:24:47.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:47.583+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:47.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:47.583+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:47.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:47.583+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:47.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:47.584+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:24:47.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:47.744+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:47.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:47.744+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:47.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:47.744+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:47.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:47.745+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:24:48.625 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:48.303+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:48.626 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:48.303+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:48.626 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:48.303+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:48.626 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:48.304+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:24:48.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:48.624+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:48.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:48.624+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:48.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:48.624+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:48.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:48.625+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:24:48.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:48.711+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:48.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:48.712+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:48.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:48.712+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:48.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:48.712+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:24:49.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:49.586+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:49.699 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:49.586+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:49.699 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:49.586+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:49.699 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:49.587+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:24:49.700 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:49.341+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:49.700 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:49.341+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:49.700 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:49.341+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:49.700 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:49.342+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:24:50.196 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:49.723+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:50.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:49.723+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:50.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:49.723+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:50.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:49.724+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:24:50.616 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:50.347+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:50.616 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:50.347+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:50.616 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:50.347+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:50.616 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:50.348+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:24:50.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:50.615+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:50.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:50.615+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:50.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:50.615+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:50.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:50.616+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:24:50.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:50.736+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:50.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:50.736+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:50.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:50.736+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:50.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:50.736+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:24:51.446 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:24:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:24:51.125+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 4133 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:24:51.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:51.328+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:51.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:51.328+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:51.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:51.328+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:51.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:51.329+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:24:51.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:51.613+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:51.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:51.613+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:51.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:51.613+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:51.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:51.614+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:24:51.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:51.757+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:51.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:51.757+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:51.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:51.757+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:51.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:51.757+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:24:52.663 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:52.309+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:52.663 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:52.309+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:52.663 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:52.309+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:52.663 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:52.310+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:24:52.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:52.662+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:52.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:52.662+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:52.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:52.662+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:52.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:52.663+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:24:52.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:52.741+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:52.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:52.741+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:52.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:52.741+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:52.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:52.742+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:24:53.657 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:53.356+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:53.657 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:53.356+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:53.657 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:53.356+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:53.657 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:53.357+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:24:53.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:53.656+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:53.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:53.656+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:53.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:53.656+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:53.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:53.657+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:24:53.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:53.778+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:53.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:53.778+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:53.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:53.778+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:53.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:53.779+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:24:54.674 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:54.328+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:54.674 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:54.328+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:54.675 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:54.328+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:54.675 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:54.329+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:24:54.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:54.673+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:54.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:54.673+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:54.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:54.673+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:54.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:54.674+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:24:54.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:54.731+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:54.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:54.731+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:54.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:54.731+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:54.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:54.732+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:24:55.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:55.285+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:55.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:55.285+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:55.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:55.285+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:55.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:55.286+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:24:56.127 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:55.714+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:56.127 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:55.714+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:56.127 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:55.714+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:56.127 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:55.715+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:24:56.128 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:55.726+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:56.128 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:55.726+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:56.128 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:55.726+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:56.128 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:55.727+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:24:56.447 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:24:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:24:56.125+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 4231 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:24:56.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:56.262+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:56.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:56.262+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:56.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:56.262+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:56.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:56.263+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:24:57.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:56.750+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:57.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:56.750+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:57.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:56.750+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:57.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:56.751+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:24:57.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:56.729+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:57.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:56.729+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:57.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:56.729+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:57.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:56.730+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:24:57.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:57.300+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:57.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:57.300+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:57.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:57.300+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:57.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:57.300+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:24:58.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:57.754+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:58.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:57.754+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:58.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:57.754+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:58.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:57.755+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:24:58.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:57.706+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:58.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:57.706+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:58.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:57.706+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:58.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:57.706+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:24:58.687 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:58.335+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:58.687 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:58.335+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:58.687 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:58.335+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:58.687 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:58.336+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:24:58.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:58.686+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:58.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:58.686+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:58.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:58.686+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:24:58.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:58.687+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:24:58.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:58.801+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:58.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:58.801+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:58.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:58.801+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:24:58.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:58.802+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:24:59.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:59.364+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:59.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:59.364+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:59.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:59.364+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:24:59.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:24:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:24:59.364+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:25:00.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:59.782+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:00.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:59.782+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:00.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:59.782+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:00.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:24:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:24:59.782+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:25:00.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:59.711+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:00.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:59.711+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:00.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:59.711+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:00.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:24:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:24:59.712+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:25:00.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:00.379+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:00.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:00.379+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:00.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:00.379+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:00.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:00.380+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:25:01.127 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:00.788+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:01.128 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:00.788+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:01.128 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:00.788+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:01.128 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:00.788+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:25:01.128 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:00.747+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:01.128 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:00.747+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:01.129 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:00.747+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:01.129 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:00.748+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:25:01.446 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:25:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:25:01.126+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 4326 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:25:01.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:01.374+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:01.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:01.374+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:01.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:01.374+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:01.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:01.375+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:25:02.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:01.790+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:02.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:01.790+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:02.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:01.790+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:02.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:01.791+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:25:02.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:01.810+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:02.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:01.810+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:02.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:01.810+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:02.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:01.811+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:25:02.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:02.376+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:02.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:02.376+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:02.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:02.376+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:02.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:02.377+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:25:03.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:02.791+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:03.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:02.791+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:03.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:02.791+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:03.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:02.791+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:25:03.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:02.778+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:03.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:02.778+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:03.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:02.778+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:03.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:02.779+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:25:03.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:03.331+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:03.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:03.331+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:03.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:03.331+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:03.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:03.332+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:25:04.168 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:03.810+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:04.168 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:03.810+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:04.168 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:03.810+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:04.168 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:03.811+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:25:04.169 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:03.818+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:04.169 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:03.818+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:04.169 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:03.818+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:04.169 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:03.819+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:25:04.450 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:04.300+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:04.450 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:04.300+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:04.451 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:04.300+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:04.451 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:04.301+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:25:05.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:04.778+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:05.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:04.778+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:05.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:04.778+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:05.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:04.779+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:25:05.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:04.808+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:05.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:04.808+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:05.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:04.808+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:05.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:04.809+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:25:05.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:05.272+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:05.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:05.272+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:05.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:05.272+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:05.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:05.273+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:25:06.128 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:05.851+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:06.128 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:05.851+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:06.129 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:05.851+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:06.129 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:05.852+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:25:06.129 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:05.795+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:06.129 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:05.795+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:06.129 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:05.795+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:06.129 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:05.795+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:25:06.428 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:25:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:25:06.127+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 4426 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:25:06.429 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:06.234+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:06.429 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:06.234+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:06.429 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:06.234+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:06.429 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:06.235+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:25:07.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:06.865+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:07.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:06.865+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:07.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:06.865+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:07.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:06.866+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:25:07.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:06.840+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:07.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:06.840+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:07.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:06.840+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:07.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:06.841+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:25:07.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:07.212+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:07.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:07.212+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:07.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:07.212+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:07.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:07.212+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:25:08.189 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:07.870+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:08.189 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:07.870+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:08.189 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:07.870+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:08.189 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:07.870+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:25:08.190 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:07.883+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:08.190 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:07.883+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:08.190 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:07.883+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:08.190 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:07.884+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:25:08.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:08.188+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:08.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:08.188+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:08.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:08.188+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:08.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:08.189+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:25:09.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:08.856+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:09.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:08.856+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:09.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:08.856+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:09.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:08.856+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:25:09.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:08.845+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:09.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:08.845+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:09.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:08.845+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:09.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:08.846+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:25:09.500 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:09.216+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:09.500 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:09.216+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:09.500 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:09.216+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:09.500 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:09.217+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:25:10.196 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:09.894+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:10.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:09.894+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:10.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:09.894+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:10.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:09.894+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:25:10.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:09.885+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:10.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:09.885+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:10.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:09.885+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:10.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:09.886+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:25:10.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:10.223+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:10.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:10.223+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:10.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:10.223+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:10.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:10.224+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:25:11.129 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:10.936+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:11.129 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:10.936+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:11.130 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:10.936+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:11.130 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:10.937+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:25:11.130 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:10.869+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:11.130 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:10.869+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:11.130 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:10.869+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:11.130 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:10.870+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:25:11.447 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:25:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:25:11.128+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 4519 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:25:11.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:11.265+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:11.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:11.265+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:11.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:11.265+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:11.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:11.265+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:25:12.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:11.857+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:12.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:11.857+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:12.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:11.857+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:12.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:11.858+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:25:12.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:11.948+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:12.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:11.948+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:12.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:11.948+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:12.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:11.949+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:25:12.573 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:12.240+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:12.573 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:12.240+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:12.573 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:12.240+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:12.573 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:12.240+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:25:13.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:12.886+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:13.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:12.886+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:13.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:12.886+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:13.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:12.887+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:25:13.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:12.935+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:13.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:12.935+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:13.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:12.935+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:13.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:12.936+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:25:13.540 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:13.249+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:13.540 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:13.249+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:13.540 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:13.249+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:13.540 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:13.250+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:25:14.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:13.959+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:14.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:13.959+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:14.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:13.959+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:14.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:13.960+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:25:14.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:13.897+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:14.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:13.897+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:14.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:13.897+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:14.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:13.898+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:25:14.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:14.286+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:14.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:14.286+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:14.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:14.286+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:14.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:14.287+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:25:15.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:14.947+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:15.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:14.947+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:15.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:14.947+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:15.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:14.948+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:25:15.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:14.902+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:15.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:14.902+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:15.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:14.902+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:15.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:14.903+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:25:15.643 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:15.242+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:15.643 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:15.242+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:15.643 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:15.242+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:15.643 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:15.243+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:25:16.197 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:25:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:25:16.129+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 4622 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:25:16.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:15.918+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:16.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:15.918+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:16.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:15.918+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:16.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:15.919+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:25:16.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:15.920+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:16.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:15.920+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:16.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:15.920+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:16.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:15.921+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:25:16.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:16.246+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:16.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:16.246+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:16.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:16.246+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:16.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:16.247+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:25:17.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:16.893+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:17.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:16.893+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:17.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:16.893+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:17.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:16.893+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:25:17.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:16.949+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:17.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:16.949+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:17.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:16.949+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:17.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:16.949+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:25:17.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:17.275+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:17.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:17.275+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:17.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:17.275+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:17.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:17.276+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:25:18.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:17.883+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:18.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:17.883+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:18.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:17.883+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:18.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:17.884+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:25:18.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:17.976+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:18.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:17.976+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:18.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:17.976+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:18.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:17.977+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:25:18.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:18.294+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:18.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:18.294+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:18.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:18.294+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:18.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:18.295+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:25:19.019 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:18.863+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:19.019 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:18.863+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:19.019 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:18.863+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:19.020 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:18.864+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:25:19.301 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:19.019+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:19.301 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:19.019+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:19.301 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:19.019+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:19.302 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:19.020+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:25:19.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:19.301+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:19.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:19.301+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:19.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:19.301+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:19.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:19.302+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:25:20.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:20.017+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:20.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:20.017+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:20.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:20.017+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:20.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:20.018+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:25:20.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:19.879+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:20.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:19.879+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:20.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:19.879+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:20.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:19.880+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:25:20.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:20.281+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:20.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:20.281+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:20.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:20.281+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:20.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:20.282+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:25:21.153 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:25:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:25:21.130+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 4724 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:25:21.153 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:20.901+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:21.154 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:20.901+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:21.154 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:20.901+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:21.154 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:20.903+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:25:21.154 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:20.989+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:21.154 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:20.989+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:21.154 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:20.989+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:21.155 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:20.990+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:25:21.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:21.323+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:21.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:21.323+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:21.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:21.323+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:21.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:21.324+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:25:22.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:21.866+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:22.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:21.866+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:22.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:21.866+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:22.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:21.867+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:25:22.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:21.947+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:22.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:21.947+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:22.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:21.948+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:22.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:21.949+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:25:22.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:22.308+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:22.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:22.308+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:22.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:22.308+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:22.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:22.308+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:25:23.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:22.861+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:23.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:22.861+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:23.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:22.861+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:23.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:22.862+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:25:23.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:22.911+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:23.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:22.911+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:23.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:22.911+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:23.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:22.911+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:25:23.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:23.313+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:23.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:23.313+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:23.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:23.313+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:23.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:23.313+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:25:24.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:23.864+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:24.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:23.864+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:24.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:23.864+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:24.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:23.865+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:25:24.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:23.894+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:24.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:23.894+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:24.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:23.894+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:24.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:23.895+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:25:24.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:24.283+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:24.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:24.283+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:24.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:24.283+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:24.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:24.284+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:25:25.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:24.900+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:25.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:24.900+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:25.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:24.900+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:25.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:24.901+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:25:25.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:24.912+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:25.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:24.912+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:25.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:24.913+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:25.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:24.914+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:25:25.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:25.264+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:25.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:25.264+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:25.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:25.264+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:25.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:25.265+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:25:26.197 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:25:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:25:26.130+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 4821 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:25:26.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:25.953+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:26.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:25.953+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:26.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:25.953+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:26.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:25.954+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:25:26.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:25.931+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:26.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:25.931+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:26.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:25.931+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:26.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:25.932+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:25:26.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:26.240+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:26.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:26.240+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:26.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:26.240+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:26.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:26.241+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:25:27.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:26.930+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:27.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:26.930+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:27.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:26.930+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:27.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:26.931+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:25:27.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:26.954+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:27.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:26.954+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:27.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:26.954+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:27.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:26.955+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:25:27.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:27.240+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:27.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:27.240+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:27.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:27.240+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:27.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:27.241+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:25:28.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:27.960+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:28.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:27.960+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:28.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:27.960+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:28.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:27.961+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:25:28.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:27.951+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:28.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:27.951+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:28.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:27.951+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:28.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:27.951+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:25:28.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:28.207+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:28.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:28.207+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:28.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:28.207+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:28.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:28.208+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:25:29.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:28.982+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:29.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:28.982+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:29.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:28.982+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:29.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:28.983+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:25:29.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:28.939+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:29.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:28.939+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:29.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:28.939+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:29.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:28.940+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:25:29.199 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:29.187+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:29.199 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:29.187+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:29.199 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:29.187+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:29.199 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:29.188+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:25:30.175 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:29.989+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:30.175 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:29.989+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:30.176 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:29.989+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:30.176 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:29.990+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:25:30.176 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:30.019+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:30.176 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:30.019+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:30.176 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:30.020+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:30.176 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:30.021+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:25:30.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:30.174+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:30.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:30.174+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:30.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:30.174+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:30.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:30.175+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:25:31.258 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:25:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:25:31.131+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 4917 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:25:31.258 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:31.150+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:31.258 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:31.150+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:31.258 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:31.150+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:31.259 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:31.151+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:25:31.259 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:31.046+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:31.259 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:31.046+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:31.259 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:31.046+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:31.260 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:31.047+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:25:31.260 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:30.946+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:31.260 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:30.946+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:31.260 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:30.946+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:31.260 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:30.947+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:25:32.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:32.195+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:32.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:32.195+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:32.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:32.195+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:32.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:32.195+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:25:32.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:32.004+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:32.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:32.004+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:32.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:32.004+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:32.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:32.004+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:25:32.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:31.976+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:32.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:31.976+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:32.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:31.976+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:32.449 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:31.977+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:25:33.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:33.189+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:33.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:33.189+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:33.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:33.189+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:33.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:33.190+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:25:33.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:32.989+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:33.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:32.989+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:33.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:32.989+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:33.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:32.990+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:25:33.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:33.012+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:33.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:33.012+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:33.449 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:33.012+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:33.449 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:33.013+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:25:34.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:34.179+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:34.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:34.179+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:34.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:34.179+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:34.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:34.179+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:25:34.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:33.984+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:34.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:33.984+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:34.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:33.984+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:34.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:33.985+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:25:34.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:34.017+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:34.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:34.017+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:34.449 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:34.017+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:34.449 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:34.018+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:25:35.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:35.134+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:35.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:35.134+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:35.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:35.134+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:35.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:35.134+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:25:35.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:34.966+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:35.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:34.966+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:35.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:34.966+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:35.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:34.967+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:25:35.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:35.046+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:35.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:35.046+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:35.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:35.046+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:35.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:35.047+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:25:36.256 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:35.975+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:36.256 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:35.975+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:36.256 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:35.975+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:36.256 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:35.976+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:25:36.257 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:36.070+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:36.257 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:36.070+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:36.257 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:36.070+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:36.257 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:36.070+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:25:36.258 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:25:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:25:36.132+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 5029 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:25:36.258 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:36.108+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:36.258 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:36.108+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:36.258 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:36.108+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:36.258 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:36.109+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:25:37.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:37.157+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:37.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:37.157+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:37.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:37.157+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:37.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:37.158+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:25:37.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:36.993+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:37.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:36.993+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:37.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:36.993+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:37.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:36.994+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:25:37.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:37.117+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:37.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:37.117+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:37.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:37.117+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:37.449 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:37.118+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:25:38.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:38.144+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:38.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:38.144+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:38.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:38.144+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:38.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:38.144+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:25:38.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:38.119+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:38.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:38.119+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:38.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:38.119+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:38.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:38.120+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:25:38.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:38.013+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:38.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:38.013+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:38.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:38.013+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:38.449 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:38.014+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:25:39.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:39.126+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:39.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:39.126+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:39.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:39.126+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:39.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:39.127+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:25:39.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:38.966+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:39.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:38.966+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:39.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:38.966+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:39.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:38.966+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:25:39.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:39.106+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:39.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:39.106+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:39.449 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:39.106+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:39.449 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:39.106+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:25:40.111 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:39.935+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:40.111 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:39.935+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:40.111 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:39.935+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:40.111 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:39.936+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:25:40.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:40.137+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:40.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:40.137+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:40.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:40.137+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:40.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:40.138+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:25:40.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:40.110+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:40.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:40.110+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:40.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:40.110+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:40.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:40.111+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:25:41.196 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:25:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:25:41.133+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 5123 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:25:41.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:40.928+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:41.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:40.928+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:41.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:40.928+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:41.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:40.929+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:25:41.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:41.067+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:41.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:41.067+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:41.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:41.067+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:41.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:41.068+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:25:41.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:41.175+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:41.199 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:41.175+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:41.199 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:41.175+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:41.199 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:41.176+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:25:42.171 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:41.882+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:42.171 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:41.882+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:42.172 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:41.882+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:42.172 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:41.882+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:25:42.172 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:42.044+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:42.172 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:42.044+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:42.172 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:42.044+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:42.172 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:42.045+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:25:42.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:42.170+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:42.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:42.170+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:42.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:42.170+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:42.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:42.171+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:25:43.166 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:43.075+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:43.166 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:43.075+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:43.167 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:43.075+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:43.167 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:43.076+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:25:43.167 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:42.837+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:43.167 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:42.837+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:43.167 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:42.837+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:43.167 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:42.838+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:25:43.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:43.165+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:43.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:43.165+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:43.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:43.165+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:43.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:43.166+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:25:43.792 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:43.791+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:43.792 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:43.791+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:43.793 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:43.791+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:44.070 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:43.792+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:25:44.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:44.069+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:44.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:44.069+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:44.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:44.069+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:44.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:44.070+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:25:44.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:44.177+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:44.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:44.177+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:44.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:44.177+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:44.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:44.178+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:25:45.107 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:44.828+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:45.108 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:44.828+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:45.108 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:44.828+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:45.108 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:44.830+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:25:45.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:45.191+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:45.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:45.191+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:45.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:45.191+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:45.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:45.192+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:25:45.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:45.106+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:45.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:45.106+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:45.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:45.106+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:45.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:45.107+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:25:46.135 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:45.873+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:46.135 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:45.873+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:46.136 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:45.873+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:46.136 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:45.874+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:25:46.136 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:46.095+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:46.136 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:46.095+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:46.136 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:46.095+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:46.136 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:46.096+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:25:46.446 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:25:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:25:46.134+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 5223 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:25:46.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:46.173+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:46.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:46.173+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:46.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:46.173+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:46.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:46.174+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:25:47.124 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:46.863+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:47.124 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:46.863+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:47.124 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:46.863+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:47.124 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:46.863+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:25:47.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:47.223+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:47.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:47.223+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:47.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:47.223+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:47.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:47.223+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:25:47.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:47.123+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:47.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:47.123+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:47.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:47.123+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:47.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:47.124+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:25:48.162 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:47.832+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:48.163 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:47.832+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:48.163 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:47.832+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:48.163 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:47.832+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:25:48.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:48.184+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:48.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:48.184+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:48.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:48.184+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:48.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:48.184+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:25:48.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:48.161+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:48.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:48.161+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:48.449 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:48.161+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:48.449 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:48.162+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:25:49.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:48.869+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:49.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:48.869+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:49.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:48.869+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:49.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:48.870+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:25:49.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:49.113+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:49.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:49.113+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:49.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:49.113+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:49.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:49.114+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:25:49.625 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:49.209+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:49.626 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:49.209+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:49.626 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:49.209+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:49.626 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:49.210+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:25:49.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:49.865+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:49.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:49.865+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:49.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:49.865+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:49.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:49.865+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:25:50.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:50.065+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:50.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:50.065+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:50.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:50.065+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:50.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:50.066+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:25:50.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:50.196+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:50.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:50.196+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:50.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:50.196+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:50.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:50.197+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:25:51.116 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:50.827+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:51.117 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:50.827+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:51.117 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:50.827+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:51.117 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:50.828+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:25:51.447 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:25:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:25:51.135+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 5338 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:25:51.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:51.115+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:51.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:51.115+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:51.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:51.115+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:51.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:51.116+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:25:51.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:51.189+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:51.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:51.189+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:51.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:51.189+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:51.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:51.190+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:25:52.121 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:51.840+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:52.121 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:51.840+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:52.121 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:51.840+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:52.121 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:51.841+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:25:52.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:52.119+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:52.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:52.119+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:52.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:52.119+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:52.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:52.120+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:25:52.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:52.166+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:52.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:52.166+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:52.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:52.166+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:52.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:52.167+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:25:53.103 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:52.816+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:53.103 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:52.816+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:53.103 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:52.816+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:53.103 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:52.817+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:25:53.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:53.202+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:53.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:53.202+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:53.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:53.202+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:53.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:53.202+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:25:53.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:53.102+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:53.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:53.102+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:53.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:53.102+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:53.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:53.103+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:25:54.077 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:53.810+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:54.077 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:53.810+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:54.077 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:53.810+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:54.078 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:53.811+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:25:54.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:54.191+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:54.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:54.191+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:54.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:54.191+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:54.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:54.191+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:25:54.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:54.076+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:54.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:54.076+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:54.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:54.076+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:54.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:54.077+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:25:54.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:54.808+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:54.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:54.808+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:54.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:54.808+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:54.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:54.809+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:25:55.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:55.174+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:55.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:55.174+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:55.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:55.174+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:55.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:55.175+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:25:55.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:55.115+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:55.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:55.115+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:55.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:55.115+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:55.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:55.116+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:25:56.138 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:55.847+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:56.138 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:55.847+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:56.138 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:55.847+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:56.138 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:55.848+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:25:56.447 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:25:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:25:56.136+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 5459 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:25:56.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:56.197+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:56.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:56.197+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:56.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:56.197+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:56.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:56.197+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:25:56.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:56.159+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:56.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:56.159+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:56.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:56.159+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:56.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:56.160+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:25:57.165 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:56.809+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:57.165 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:56.809+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:57.166 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:56.809+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:57.166 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:56.810+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:25:57.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:57.164+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:57.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:57.164+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:57.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:57.164+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:57.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:57.165+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:25:57.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:57.195+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:57.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:57.195+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:57.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:57.195+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:57.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:57.196+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:25:58.169 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:57.833+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:58.170 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:57.833+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:58.170 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:57.833+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:58.170 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:57.833+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:25:58.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:58.166+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:58.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:58.166+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:58.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:58.166+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:58.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:58.167+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:25:58.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:58.207+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:58.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:58.207+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:58.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:58.207+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:58.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:58.208+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:25:59.162 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:58.821+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:59.162 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:58.821+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:59.162 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:58.821+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:25:59.162 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:58.822+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:25:59.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:59.204+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:59.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:59.204+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:59.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:59.204+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:25:59.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:25:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:25:59.205+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:25:59.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:59.161+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:59.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:59.161+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:59.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:59.161+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:25:59.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:25:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:25:59.162+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:26:00.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:59.871+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:00.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:59.871+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:00.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:59.871+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:00.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:25:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:25:59.872+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:26:00.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:00.199+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:00.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:00.199+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:00.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:00.199+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:00.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:00.199+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:26:00.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:00.243+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:00.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:00.243+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:00.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:00.243+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:00.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:00.244+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:26:01.138 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:00.881+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:01.138 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:00.881+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:01.138 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:00.881+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:01.138 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:00.882+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:26:01.447 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:26:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:26:01.137+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 5553 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:26:01.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:01.150+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:01.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:01.150+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:01.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:01.150+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:01.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:01.151+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:26:01.449 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:01.199+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:01.449 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:01.199+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:01.450 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:01.199+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:01.450 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:01.200+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:26:02.184 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:01.889+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:02.184 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:01.889+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:02.184 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:01.889+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:02.184 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:01.890+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:26:02.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:02.208+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:02.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:02.208+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:02.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:02.208+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:02.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:02.209+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:26:02.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:02.184+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:02.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:02.184+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:02.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:02.184+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:02.449 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:02.184+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:26:03.149 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:02.851+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:03.150 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:02.851+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:03.150 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:02.851+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:03.150 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:02.852+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:26:03.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:03.246+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:03.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:03.246+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:03.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:03.246+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:03.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:03.247+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:26:03.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:03.149+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:03.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:03.149+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:03.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:03.149+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:03.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:03.150+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:26:04.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:03.894+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:04.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:03.894+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:04.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:03.894+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:04.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:03.894+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:26:04.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:04.217+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:04.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:04.217+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:04.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:04.217+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:04.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:04.218+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:26:04.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:04.197+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:04.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:04.197+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:04.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:04.197+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:04.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:04.198+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:26:05.189 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:04.886+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:05.190 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:04.886+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:05.190 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:04.886+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:05.190 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:04.888+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:26:05.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:05.189+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:05.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:05.189+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:05.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:05.189+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:05.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:05.190+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:26:05.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:05.243+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:05.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:05.243+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:05.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:05.243+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:05.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:05.244+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:26:06.139 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:05.875+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:06.139 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:05.875+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:06.139 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:05.875+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:06.140 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:05.876+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:26:06.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:06.184+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:06.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:06.184+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:06.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:06.184+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:06.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:06.185+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:26:06.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:06.229+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:06.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:06.229+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:06.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:06.229+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:06.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:06.230+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:26:06.448 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:26:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:26:06.138+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 5650 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:26:07.150 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:06.835+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:07.150 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:06.835+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:07.151 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:06.835+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:07.151 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:06.836+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:26:07.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:07.278+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:07.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:07.278+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:07.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:07.278+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:07.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:07.278+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:26:07.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:07.149+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:07.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:07.149+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:07.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:07.149+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:07.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:07.150+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:26:08.146 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:07.855+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:08.146 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:07.855+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:08.147 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:07.855+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:08.147 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:07.856+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:26:08.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:08.252+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:08.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:08.252+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:08.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:08.252+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:08.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:08.253+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:26:08.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:08.145+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:08.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:08.145+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:08.449 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:08.145+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:08.449 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:08.145+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:26:09.108 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:08.836+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:09.108 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:08.836+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:09.108 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:08.836+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:09.108 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:08.837+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:26:09.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:09.276+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:09.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:09.276+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:09.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:09.276+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:09.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:09.277+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:26:09.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:09.106+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:09.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:09.106+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:09.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:09.107+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:09.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:09.108+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:26:10.146 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:09.793+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:10.146 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:09.793+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:10.147 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:09.793+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:10.147 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:09.794+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:26:10.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:10.145+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:10.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:10.145+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:10.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:10.145+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:10.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:10.146+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:26:10.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:10.296+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:10.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:10.296+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:10.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:10.296+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:10.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:10.298+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:26:11.140 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:10.777+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:11.140 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:10.777+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:11.140 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:10.777+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:11.140 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:10.778+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:26:11.447 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:26:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:26:11.138+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 5744 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:26:11.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:11.189+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:11.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:11.189+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:11.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:11.189+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:11.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:11.190+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:26:11.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:11.284+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:11.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:11.284+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:11.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:11.284+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:11.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:11.285+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:26:11.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:11.809+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:11.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:11.809+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:11.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:11.809+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:11.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:11.810+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:26:12.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:12.301+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:12.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:12.301+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:12.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:12.301+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:12.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:12.302+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:26:12.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:12.174+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:12.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:12.174+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:12.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:12.174+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:12.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:12.174+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:26:13.143 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:12.858+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:13.143 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:12.859+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:13.143 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:12.859+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:13.143 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:12.860+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:26:13.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:13.140+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:13.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:13.140+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:13.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:13.140+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:13.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:13.141+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:26:13.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:13.319+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:13.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:13.319+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:13.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:13.319+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:13.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:13.320+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:26:14.177 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:13.859+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:14.178 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:13.859+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:14.178 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:13.859+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:14.178 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:13.860+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:26:14.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:14.332+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:14.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:14.332+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:14.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:14.332+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:14.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:14.333+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:26:14.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:14.176+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:14.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:14.176+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:14.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:14.176+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:14.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:14.177+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:26:15.171 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:14.839+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:15.171 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:14.839+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:15.172 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:14.839+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:15.172 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:14.840+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:26:15.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:15.305+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:15.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:15.305+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:15.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:15.305+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:15.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:15.305+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:26:15.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:15.170+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:15.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:15.170+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:15.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:15.170+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:15.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:15.170+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:26:16.141 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:15.865+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:16.141 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:15.865+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:16.141 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:15.865+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:16.141 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:15.866+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:26:16.446 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:26:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:26:16.139+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 5841 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:26:16.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:16.215+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:16.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:16.215+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:16.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:16.215+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:16.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:16.216+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:26:16.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:16.326+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:16.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:16.326+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:16.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:16.326+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:16.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:16.327+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:26:17.188 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:16.896+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:17.188 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:16.896+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:17.188 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:16.896+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:17.188 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:16.897+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:26:17.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:17.186+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:17.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:17.186+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:17.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:17.186+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:17.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:17.187+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:26:17.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:17.359+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:17.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:17.359+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:17.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:17.359+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:17.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:17.360+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:26:18.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:17.908+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:18.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:17.908+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:18.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:17.908+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:18.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:17.908+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:26:18.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:18.215+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:18.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:18.215+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:18.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:18.215+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:18.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:18.216+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:26:18.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:18.359+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:18.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:18.359+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:18.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:18.359+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:18.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:18.359+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:26:19.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:18.934+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:19.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:18.934+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:19.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:18.934+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:19.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:18.935+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:26:19.614 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:19.206+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:19.615 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:19.206+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:19.615 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:19.206+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:19.615 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:19.207+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:26:19.615 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:19.386+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:19.615 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:19.386+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:19.616 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:19.386+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:19.616 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:19.387+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:26:19.616 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-05-04T12:26:19.617 DEBUG:tasks.cephfs.mount:Force/lazy unmounting on client.0 2024-05-04T12:26:19.617 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-04T12:26:19.618 DEBUG:teuthology.orchestra.run.smithi079:> sudo umount --lazy --force /home/ubuntu/cephtest/mnt.0 2024-05-04T12:26:19.658 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests: Cannot mkdir: Input/output error 2024-05-04T12:26:19.658 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/relabel: Cannot mkdir: Input/output error 2024-05-04T12:26:19.658 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/relabel/Makefile: Cannot open: Input/output error 2024-05-04T12:26:19.659 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/relabel/selinux_relabel.sh: Cannot open: Input/output error 2024-05-04T12:26:19.659 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/capable_file: Cannot mkdir: Input/output error 2024-05-04T12:26:19.659 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/capable_file/selinux_lease.c: Cannot open: Input/output error 2024-05-04T12:26:19.659 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/capable_file/Makefile: Cannot open: Input/output error 2024-05-04T12:26:19.659 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/capable_file/selinux_capable_file.sh: Cannot open: Input/output error 2024-05-04T12:26:19.659 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/sem: Cannot mkdir: Input/output error 2024-05-04T12:26:19.659 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/sem/selinux_semget.c: Cannot open: Input/output error 2024-05-04T12:26:19.659 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/sem/selinux_sem.sh: Cannot open: Input/output error 2024-05-04T12:26:19.660 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/sem/selinux_getinfo.c: Cannot open: Input/output error 2024-05-04T12:26:19.660 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/sem/Makefile: Cannot open: Input/output error 2024-05-04T12:26:19.660 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/sem/selinux_semop.c: Cannot open: Input/output error 2024-05-04T12:26:19.660 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/sem/selinux_semctl.c: Cannot open: Input/output error 2024-05-04T12:26:19.660 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/shm: Cannot mkdir: Input/output error 2024-05-04T12:26:19.660 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/shm/selinux_shmctl.c: Cannot open: Input/output error 2024-05-04T12:26:19.660 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/shm/Makefile: Cannot open: Input/output error 2024-05-04T12:26:19.660 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/shm/selinux_shmget.c: Cannot open: Input/output error 2024-05-04T12:26:19.661 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/shm/selinux_shm.sh: Cannot open: Input/output error 2024-05-04T12:26:19.661 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/shm/selinux_shmat.c: Cannot open: Input/output error 2024-05-04T12:26:19.661 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/rename: Cannot mkdir: Input/output error 2024-05-04T12:26:19.661 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/rename/Makefile: Cannot open: Input/output error 2024-05-04T12:26:19.661 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/rename/selinux_rename.sh: Cannot open: Input/output error 2024-05-04T12:26:19.661 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/exectrace: Cannot mkdir: Input/output error 2024-05-04T12:26:19.661 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/exectrace/selinux_exectrace.sh: Cannot open: Input/output error 2024-05-04T12:26:19.662 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/exectrace/Makefile: Cannot open: Input/output error 2024-05-04T12:26:19.662 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/exectrace/selinux_exectrace_child.c: Cannot open: Input/output error 2024-05-04T12:26:19.662 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/exectrace/selinux_exectrace_parent.c: Cannot open: Input/output error 2024-05-04T12:26:19.662 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/setattr: Cannot mkdir: Input/output error 2024-05-04T12:26:19.662 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/setattr/Makefile: Cannot open: Input/output error 2024-05-04T12:26:19.662 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/setattr/selinux_setattr.sh: Cannot open: Input/output error 2024-05-04T12:26:19.663 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/ptrace: Cannot mkdir: Input/output error 2024-05-04T12:26:19.663 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/ptrace/selinux_ptrace.c: Cannot open: Input/output error 2024-05-04T12:26:19.663 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/ptrace/selinux_ptrace.sh: Cannot open: Input/output error 2024-05-04T12:26:19.663 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/ptrace/Makefile: Cannot open: Input/output error 2024-05-04T12:26:19.663 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/ptrace/selinux_ptrace_wait.sh: Cannot open: Input/output error 2024-05-04T12:26:19.663 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/task_create: Cannot mkdir: Input/output error 2024-05-04T12:26:19.663 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/task_create/selinux_task_create.sh: Cannot open: Input/output error 2024-05-04T12:26:19.663 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/task_create/Makefile: Cannot open: Input/output error 2024-05-04T12:26:19.663 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/task_create/selinux_task_create_parent.c: Cannot open: Input/output error 2024-05-04T12:26:19.664 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/execute_no_trans: Cannot mkdir: Input/output error 2024-05-04T12:26:19.664 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/execute_no_trans/Makefile: Cannot open: Input/output error 2024-05-04T12:26:19.664 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/execute_no_trans/selinux_execute_no_trans.sh: Cannot open: Input/output error 2024-05-04T12:26:19.664 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/inherit: Cannot mkdir: Input/output error 2024-05-04T12:26:19.664 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/inherit/selinux_inherit_parent.c: Cannot open: Input/output error 2024-05-04T12:26:19.664 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/inherit/selinux_inherit.sh: Cannot open: Input/output error 2024-05-04T12:26:19.664 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/inherit/Makefile: Cannot open: Input/output error 2024-05-04T12:26:19.664 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/inherit/selinux_inherit_child.c: Cannot open: Input/output error 2024-05-04T12:26:19.665 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/file: Cannot mkdir: Input/output error 2024-05-04T12:26:19.665 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/file/selinux_fcntl.c: Cannot open: Input/output error 2024-05-04T12:26:19.665 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/file/selinux_wait_io.c: Cannot open: Input/output error 2024-05-04T12:26:19.665 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/file/selinux_mmap.c: Cannot open: Input/output error 2024-05-04T12:26:19.665 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/file/Makefile: Cannot open: Input/output error 2024-05-04T12:26:19.665 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/file/selinux_seek.c: Cannot open: Input/output error 2024-05-04T12:26:19.665 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/file/selinux_lock.c: Cannot open: Input/output error 2024-05-04T12:26:19.665 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/file/selinux_sigiotask.c: Cannot open: Input/output error 2024-05-04T12:26:19.666 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/file/selinux_mprotect.c: Cannot open: Input/output error 2024-05-04T12:26:19.666 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/file/selinux_file.sh: Cannot open: Input/output error 2024-05-04T12:26:19.666 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/file/selinux_nofcntl.c: Cannot open: Input/output error 2024-05-04T12:26:19.666 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/Makefile: Cannot open: Input/output error 2024-05-04T12:26:19.666 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/task_setscheduler: Cannot mkdir: Input/output error 2024-05-04T12:26:19.666 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/task_setscheduler/selinux_task_setscheduler_source.c: Cannot open: Input/output error 2024-05-04T12:26:19.666 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/task_setscheduler/Makefile: Cannot open: Input/output error 2024-05-04T12:26:19.666 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/task_setscheduler/selinux_task_setscheduler.sh: Cannot open: Input/output error 2024-05-04T12:26:19.667 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/task_setscheduler/selinux_task_setscheduler_target.c: Cannot open: Input/output error 2024-05-04T12:26:19.667 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/mkdir: Cannot mkdir: Input/output error 2024-05-04T12:26:19.668 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/mkdir/Makefile: Cannot open: Input/output error 2024-05-04T12:26:19.668 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/mkdir/selinux_mkdir.sh: Cannot open: Input/output error 2024-05-04T12:26:19.668 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/setnice: Cannot mkdir: Input/output error 2024-05-04T12:26:19.669 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi079.front.sepia.ceph.com 2024-05-04T12:26:19.669 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-04T12:26:19.669 DEBUG:teuthology.orchestra.run.smithi079:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-05-04T12:26:19.693 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-05-04T12:26:19.693 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/setnice/Makefile: Cannot open: Input/output error 2024-05-04T12:26:19.694 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/setnice/selinux_setnice_parent.c: Cannot open: Input/output error 2024-05-04T12:26:19.694 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/setnice/selinux_setnice_child.c: Cannot open: Input/output error 2024-05-04T12:26:19.694 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/setnice/selinux_setnice.sh: Cannot open: Input/output error 2024-05-04T12:26:19.694 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/stat: Cannot mkdir: Input/output error 2024-05-04T12:26:19.694 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/stat/Makefile: Cannot open: Input/output error 2024-05-04T12:26:19.694 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/stat/selinux_stat.sh: Cannot open: Input/output error 2024-05-04T12:26:19.694 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/dyntrace: Cannot mkdir: Input/output error 2024-05-04T12:26:19.694 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/dyntrace/Makefile: Cannot open: Input/output error 2024-05-04T12:26:19.695 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/dyntrace/selinux_dyntrace_parent.c: Cannot open: Input/output error 2024-05-04T12:26:19.695 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/dyntrace/selinux_dyntrace.sh: Cannot open: Input/output error 2024-05-04T12:26:19.695 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/dyntrace/selinux_dyntrace_child.c: Cannot open: Input/output error 2024-05-04T12:26:19.695 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/task_setpgid: Cannot mkdir: Input/output error 2024-05-04T12:26:19.695 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/task_setpgid/Makefile: Cannot open: Input/output error 2024-05-04T12:26:19.695 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/task_setpgid/selinux_task_setpgid.sh: Cannot open: Input/output error 2024-05-04T12:26:19.695 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/task_setpgid/selinux_task_setpgid_source.c: Cannot open: Input/output error 2024-05-04T12:26:19.695 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/capable_net: Cannot mkdir: Input/output error 2024-05-04T12:26:19.696 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/capable_net/selinux_capable_net.sh: Cannot open: Input/output error 2024-05-04T12:26:19.696 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/capable_net/Makefile: Cannot open: Input/output error 2024-05-04T12:26:19.696 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/capable_net/selinux_bind.c: Cannot open: Input/output error 2024-05-04T12:26:19.696 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/capable_net/selinux_raw.c: Cannot open: Input/output error 2024-05-04T12:26:19.696 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/runtest.sh: Cannot open: Input/output error 2024-05-04T12:26:19.696 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/capable_sys: Cannot mkdir: Input/output error 2024-05-04T12:26:19.696 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/capable_sys/selinux_chroot.c: Cannot open: Input/output error 2024-05-04T12:26:19.696 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/capable_sys/selinux_capable_sys.sh: Cannot open: Input/output error 2024-05-04T12:26:19.697 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/capable_sys/Makefile: Cannot open: Input/output error 2024-05-04T12:26:19.697 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/capable_sys/selinux_rawio.c: Cannot open: Input/output error 2024-05-04T12:26:19.697 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/capable_sys/selinux_nice.c: Cannot open: Input/output error 2024-05-04T12:26:19.697 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/capable_sys/selinux_hostname.c: Cannot open: Input/output error 2024-05-04T12:26:19.697 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/task_getpgid: Cannot mkdir: Input/output error 2024-05-04T12:26:19.697 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/task_getpgid/selinux_task_getpgid.sh: Cannot open: Input/output error 2024-05-04T12:26:19.697 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/task_getpgid/Makefile: Cannot open: Input/output error 2024-05-04T12:26:19.697 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/task_getpgid/selinux_task_getpgid_target.c: Cannot open: Input/output error 2024-05-04T12:26:19.698 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/task_getpgid/selinux_task_getpgid_source.c: Cannot open: Input/output error 2024-05-04T12:26:19.698 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/entrypoint: Cannot mkdir: Input/output error 2024-05-04T12:26:19.698 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/entrypoint/selinux_entrypoint.sh: Cannot open: Input/output error 2024-05-04T12:26:19.698 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/entrypoint/Makefile: Cannot open: Input/output error 2024-05-04T12:26:19.698 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/open: Cannot mkdir: Input/output error 2024-05-04T12:26:19.698 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/open/Makefile: Cannot open: Input/output error 2024-05-04T12:26:19.698 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/open/selinux_fopen.c: Cannot open: Input/output error 2024-05-04T12:26:19.698 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/open/selinux_open.sh: Cannot open: Input/output error 2024-05-04T12:26:19.698 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/open/selinux_append2write.c: Cannot open: Input/output error 2024-05-04T12:26:19.699 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/dyntrans: Cannot mkdir: Input/output error 2024-05-04T12:26:19.699 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/dyntrans/Makefile: Cannot open: Input/output error 2024-05-04T12:26:19.699 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/dyntrans/selinux_dyntrans_parent.c: Cannot open: Input/output error 2024-05-04T12:26:19.699 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/dyntrans/selinux_dyntrans.sh: Cannot open: Input/output error 2024-05-04T12:26:19.699 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/sysctl: Cannot mkdir: Input/output error 2024-05-04T12:26:19.699 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/sysctl/selinux_sysctl.sh: Cannot open: Input/output error 2024-05-04T12:26:19.699 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/sysctl/Makefile: Cannot open: Input/output error 2024-05-04T12:26:19.699 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/domain_trans: Cannot mkdir: Input/output error 2024-05-04T12:26:19.700 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/domain_trans/Makefile: Cannot open: Input/output error 2024-05-04T12:26:19.700 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/domain_trans/selinux_domain_trans.sh: Cannot open: Input/output error 2024-05-04T12:26:19.700 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/wait: Cannot mkdir: Input/output error 2024-05-04T12:26:19.700 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/wait/selinux_wait_parent.c: Cannot open: Input/output error 2024-05-04T12:26:19.700 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/wait/Makefile: Cannot open: Input/output error 2024-05-04T12:26:19.700 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/wait/selinux_wait_child.c: Cannot open: Input/output error 2024-05-04T12:26:19.700 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/wait/selinux_wait.sh: Cannot open: Input/output error 2024-05-04T12:26:19.700 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/readlink: Cannot mkdir: Input/output error 2024-05-04T12:26:19.700 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/readlink/Makefile: Cannot open: Input/output error 2024-05-04T12:26:19.701 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/readlink/selinux_readlink.sh: Cannot open: Input/output error 2024-05-04T12:26:19.701 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/msg: Cannot mkdir: Input/output error 2024-05-04T12:26:19.701 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/msg/selinux_msg.sh: Cannot open: Input/output error 2024-05-04T12:26:19.701 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/msg/Makefile: Cannot open: Input/output error 2024-05-04T12:26:19.701 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/msg/selinux_msgctl.c: Cannot open: Input/output error 2024-05-04T12:26:19.701 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/msg/selinux_msgrcv.c: Cannot open: Input/output error 2024-05-04T12:26:19.701 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/msg/selinux_msgsnd.c: Cannot open: Input/output error 2024-05-04T12:26:19.701 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/msg/selinux_msgget.c: Cannot open: Input/output error 2024-05-04T12:26:19.702 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/sigkill: Cannot mkdir: Input/output error 2024-05-04T12:26:19.702 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/sigkill/Makefile: Cannot open: Input/output error 2024-05-04T12:26:19.702 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/sigkill/selinux_sigkill.sh: Cannot open: Input/output error 2024-05-04T12:26:19.702 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/sigkill/selinux_sigkill_server.c: Cannot open: Input/output error 2024-05-04T12:26:19.702 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/ioctl: Cannot mkdir: Input/output error 2024-05-04T12:26:19.702 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/ioctl/selinux_ioctl.sh: Cannot open: Input/output error 2024-05-04T12:26:19.702 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/ioctl/Makefile: Cannot open: Input/output error 2024-05-04T12:26:19.702 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/ioctl/selinux_noioctl.c: Cannot open: Input/output error 2024-05-04T12:26:19.702 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/ioctl/selinux_ioctl.c: Cannot open: Input/output error 2024-05-04T12:26:19.703 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/link: Cannot mkdir: Input/output error 2024-05-04T12:26:19.703 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/link/Makefile: Cannot open: Input/output error 2024-05-04T12:26:19.703 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/link/selinux_link.sh: Cannot open: Input/output error 2024-05-04T12:26:19.703 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/execshare: Cannot mkdir: Input/output error 2024-05-04T12:26:19.703 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/execshare/selinux_execshare_child.c: Cannot open: Input/output error 2024-05-04T12:26:19.703 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/execshare/Makefile: Cannot open: Input/output error 2024-05-04T12:26:19.703 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/execshare/selinux_execshare_parent.c: Cannot open: Input/output error 2024-05-04T12:26:19.703 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/execshare/selinux_execshare.sh: Cannot open: Input/output error 2024-05-04T12:26:19.704 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/task_getscheduler: Cannot mkdir: Input/output error 2024-05-04T12:26:19.704 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/task_getscheduler/selinux_task_getscheduler_target.c: Cannot open: Input/output error 2024-05-04T12:26:19.704 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/task_getscheduler/Makefile: Cannot open: Input/output error 2024-05-04T12:26:19.704 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/task_getscheduler/selinux_task_getscheduler.sh: Cannot open: Input/output error 2024-05-04T12:26:19.704 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/task_getscheduler/selinux_task_getscheduler_source.c: Cannot open: Input/output error 2024-05-04T12:26:19.704 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/fdreceive: Cannot mkdir: Input/output error 2024-05-04T12:26:19.704 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/fdreceive/Makefile: Cannot open: Input/output error 2024-05-04T12:26:19.704 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/fdreceive/selinux_fdreceive_server.c: Cannot open: Input/output error 2024-05-04T12:26:19.704 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/fdreceive/selinux_fdreceive.sh: Cannot open: Input/output error 2024-05-04T12:26:19.705 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/fdreceive/selinux_fdreceive_client.c: Cannot open: Input/output error 2024-05-04T12:26:19.705 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/task_setnice: Cannot mkdir: Input/output error 2024-05-04T12:26:19.705 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/task_setnice/Makefile: Cannot open: Input/output error 2024-05-04T12:26:19.705 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/task_setnice/selinux_task_setnice_target.c: Cannot open: Input/output error 2024-05-04T12:26:19.705 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/task_setnice/selinux_task_setnice.sh: Cannot open: Input/output error 2024-05-04T12:26:19.705 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/bounds: Cannot mkdir: Input/output error 2024-05-04T12:26:19.705 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/bounds/Makefile: Cannot open: Input/output error 2024-05-04T12:26:19.705 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/bounds/selinux_bounds.sh: Cannot open: Input/output error 2024-05-04T12:26:19.705 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/bounds/selinux_bounds_thread.c: Cannot open: Input/output error 2024-05-04T12:26:19.706 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/rxdir: Cannot mkdir: Input/output error 2024-05-04T12:26:19.706 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/rxdir/selinux_rxdir.sh: Cannot open: Input/output error 2024-05-04T12:26:19.706 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/rxdir/Makefile: Cannot open: Input/output error 2024-05-04T12:26:19.706 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/task_getsid: Cannot mkdir: Input/output error 2024-05-04T12:26:19.706 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/task_getsid/Makefile: Cannot open: Input/output error 2024-05-04T12:26:19.706 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/task_getsid/selinux_task_getsid_source.c: Cannot open: Input/output error 2024-05-04T12:26:19.706 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/task_getsid/selinux_task_getsid.sh: Cannot open: Input/output error 2024-05-04T12:26:19.706 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/task_getsid/selinux_task_getsid_target.c: Cannot open: Input/output error 2024-05-04T12:26:19.707 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/tests/Makefile.inc: Cannot open: Input/output error 2024-05-04T12:26:19.707 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite/ChangeLog: Cannot open: Input/output error 2024-05-04T12:26:19.707 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/selinux-testsuite: Cannot utime: Input/output error 2024-05-04T12:26:19.707 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/tomoyo: Cannot mkdir: Input/output error 2024-05-04T12:26:19.707 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/tomoyo/Makefile: Cannot open: Input/output error 2024-05-04T12:26:19.707 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/tomoyo/README: Cannot open: Input/output error 2024-05-04T12:26:19.707 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/tomoyo/testall.sh: Cannot open: Input/output error 2024-05-04T12:26:19.707 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/tomoyo/tomoyo_file_test.c: Cannot open: Input/output error 2024-05-04T12:26:19.707 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/tomoyo/newns.c: Cannot open: Input/output error 2024-05-04T12:26:19.708 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/tomoyo/include.h: Cannot open: Input/output error 2024-05-04T12:26:19.708 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/filecaps: Cannot mkdir: Input/output error 2024-05-04T12:26:19.708 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/filecaps/print_caps.c: Cannot open: Input/output error 2024-05-04T12:26:19.708 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/filecaps/verify_caps_exec.c: Cannot open: Input/output error 2024-05-04T12:26:19.708 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/filecaps/checkforlibcap.sh: Cannot open: Input/output error 2024-05-04T12:26:19.708 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/filecaps/Makefile: Cannot open: Input/output error 2024-05-04T12:26:19.708 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/filecaps/filecapstest.sh: Cannot open: Input/output error 2024-05-04T12:26:19.708 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/filecaps/check_simple_capset.c: Cannot open: Input/output error 2024-05-04T12:26:19.709 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/filecaps/README: Cannot open: Input/output error 2024-05-04T12:26:19.709 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/filecaps/checkforfilecaps.sh: Cannot open: Input/output error 2024-05-04T12:26:19.709 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/filecaps/inh_capped.c: Cannot open: Input/output error 2024-05-04T12:26:19.709 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/mmc_security: Cannot mkdir: Input/output error 2024-05-04T12:26:19.709 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/mmc_security/change_password.sh: Cannot open: Input/output error 2024-05-04T12:26:19.709 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/mmc_security/ask_password.sh: Cannot open: Input/output error 2024-05-04T12:26:19.709 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/mmc_security/LICENSE: Cannot open: Input/output error 2024-05-04T12:26:19.709 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/mmc_security/Makefile: Cannot open: Input/output error 2024-05-04T12:26:19.710 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/mmc_security/README: Cannot open: Input/output error 2024-05-04T12:26:19.710 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/mmc_security/AUTHORS: Cannot open: Input/output error 2024-05-04T12:26:19.710 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/mmc_security/force_erase.sh: Cannot open: Input/output error 2024-05-04T12:26:19.710 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/mmc_security/remove_password.sh: Cannot open: Input/output error 2024-05-04T12:26:19.710 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security/mmc_security/assign_password.sh: Cannot open: Input/output error 2024-05-04T12:26:19.710 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/security: Cannot utime: Input/output error 2024-05-04T12:26:19.710 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls: Cannot mkdir: Input/output error 2024-05-04T12:26:19.710 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/waitid: Cannot mkdir: Input/output error 2024-05-04T12:26:19.711 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/waitid/waitid01.c: Cannot open: Input/output error 2024-05-04T12:26:19.711 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/waitid/Makefile: Cannot open: Input/output error 2024-05-04T12:26:19.711 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/waitid/waitid02.c: Cannot open: Input/output error 2024-05-04T12:26:19.711 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/eventfd2: Cannot mkdir: Input/output error 2024-05-04T12:26:19.711 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/eventfd2/eventfd2_02.c: Cannot open: Input/output error 2024-05-04T12:26:19.711 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/eventfd2/eventfd2_01.c: Cannot open: Input/output error 2024-05-04T12:26:19.711 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/eventfd2/Makefile: Cannot open: Input/output error 2024-05-04T12:26:19.711 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/eventfd2/eventfd2_03.c: Cannot open: Input/output error 2024-05-04T12:26:19.712 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/paging: Cannot mkdir: Input/output error 2024-05-04T12:26:19.712 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/paging/Makefile: Cannot open: Input/output error 2024-05-04T12:26:19.712 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/signal: Cannot mkdir: Input/output error 2024-05-04T12:26:19.712 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/signal/signal01.c: Cannot open: Input/output error 2024-05-04T12:26:19.712 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/signal/Makefile: Cannot open: Input/output error 2024-05-04T12:26:19.712 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/signal/signal03.c: Cannot open: Input/output error 2024-05-04T12:26:19.712 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/signal/signal05.c: Cannot open: Input/output error 2024-05-04T12:26:19.712 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/signal/signal02.c: Cannot open: Input/output error 2024-05-04T12:26:19.712 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/signal/signal04.c: Cannot open: Input/output error 2024-05-04T12:26:19.713 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/settimeofday: Cannot mkdir: Input/output error 2024-05-04T12:26:19.713 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/settimeofday/settimeofday01.c: Cannot open: Input/output error 2024-05-04T12:26:19.713 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/settimeofday/settimeofday02.c: Cannot open: Input/output error 2024-05-04T12:26:19.713 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/settimeofday/Makefile: Cannot open: Input/output error 2024-05-04T12:26:19.713 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/remap_file_pages: Cannot mkdir: Input/output error 2024-05-04T12:26:19.713 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/remap_file_pages/remap_file_pages01.c: Cannot open: Input/output error 2024-05-04T12:26:19.713 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/remap_file_pages/remap_file_pages02.c: Cannot open: Input/output error 2024-05-04T12:26:19.713 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/remap_file_pages/Makefile: Cannot open: Input/output error 2024-05-04T12:26:19.714 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/io_getevents: Cannot mkdir: Input/output error 2024-05-04T12:26:19.714 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/io_getevents/io_getevents01.c: Cannot open: Input/output error 2024-05-04T12:26:19.714 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/io_getevents/Makefile: Cannot open: Input/output error 2024-05-04T12:26:19.714 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/mknod: Cannot mkdir: Input/output error 2024-05-04T12:26:19.714 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/mknod/mknod05.c: Cannot open: Input/output error 2024-05-04T12:26:19.714 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/mknod/mknod01.c: Cannot open: Input/output error 2024-05-04T12:26:19.714 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/mknod/mknod08.c: Cannot open: Input/output error 2024-05-04T12:26:19.714 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/mknod/Makefile: Cannot open: Input/output error 2024-05-04T12:26:19.714 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/mknod/mknod07.c: Cannot open: Input/output error 2024-05-04T12:26:19.715 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/mknod/mknod02.c: Cannot open: Input/output error 2024-05-04T12:26:19.715 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/mknod/mknod09.c: Cannot open: Input/output error 2024-05-04T12:26:19.715 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/mknod/mknod04.c: Cannot open: Input/output error 2024-05-04T12:26:19.715 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/mknod/mknod06.c: Cannot open: Input/output error 2024-05-04T12:26:19.715 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/mknod/mknod03.c: Cannot open: Input/output error 2024-05-04T12:26:19.715 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/close: Cannot mkdir: Input/output error 2024-05-04T12:26:19.715 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/close/close01.c: Cannot open: Input/output error 2024-05-04T12:26:19.715 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/close/close02.c: Cannot open: Input/output error 2024-05-04T12:26:19.715 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/close/Makefile: Cannot open: Input/output error 2024-05-04T12:26:19.716 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/close/close08.c: Cannot open: Input/output error 2024-05-04T12:26:19.716 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/setrlimit: Cannot mkdir: Input/output error 2024-05-04T12:26:19.716 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/setrlimit/setrlimit02.c: Cannot open: Input/output error 2024-05-04T12:26:19.716 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/setrlimit/Makefile: Cannot open: Input/output error 2024-05-04T12:26:19.716 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/setrlimit/setrlimit01.c: Cannot open: Input/output error 2024-05-04T12:26:19.716 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/setrlimit/setrlimit03.c: Cannot open: Input/output error 2024-05-04T12:26:19.716 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/setdomainname: Cannot mkdir: Input/output error 2024-05-04T12:26:19.716 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/setdomainname/setdomainname01.c: Cannot open: Input/output error 2024-05-04T12:26:19.717 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/setdomainname/setdomainname03.c: Cannot open: Input/output error 2024-05-04T12:26:19.717 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/setdomainname/Makefile: Cannot open: Input/output error 2024-05-04T12:26:19.717 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/setdomainname/setdomainname02.c: Cannot open: Input/output error 2024-05-04T12:26:19.717 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/writev: Cannot mkdir: Input/output error 2024-05-04T12:26:19.717 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/writev/writev04.c: Cannot open: Input/output error 2024-05-04T12:26:19.717 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/writev/writev05.c: Cannot open: Input/output error 2024-05-04T12:26:19.717 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/writev/Makefile: Cannot open: Input/output error 2024-05-04T12:26:19.717 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/writev/writev06.c: Cannot open: Input/output error 2024-05-04T12:26:19.717 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/writev/writev01.c: Cannot open: Input/output error 2024-05-04T12:26:19.718 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/writev/writev03.c: Cannot open: Input/output error 2024-05-04T12:26:19.718 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/writev/writev02.c: Cannot open: Input/output error 2024-05-04T12:26:19.718 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/pause: Cannot mkdir: Input/output error 2024-05-04T12:26:19.718 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/pause/Makefile: Cannot open: Input/output error 2024-05-04T12:26:19.718 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/pause/pause01.c: Cannot open: Input/output error 2024-05-04T12:26:19.718 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/pause/pause02.c: Cannot open: Input/output error 2024-05-04T12:26:19.718 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/pause/pause03.c: Cannot open: Input/output error 2024-05-04T12:26:19.718 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/mincore: Cannot mkdir: Input/output error 2024-05-04T12:26:19.719 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/mincore/mincore02.c: Cannot open: Input/output error 2024-05-04T12:26:19.719 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/mincore/Makefile: Cannot open: Input/output error 2024-05-04T12:26:19.719 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/mincore/mincore01.c: Cannot open: Input/output error 2024-05-04T12:26:19.719 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/eventfd: Cannot mkdir: Input/output error 2024-05-04T12:26:19.719 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/eventfd/Makefile: Cannot open: Input/output error 2024-05-04T12:26:19.719 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/eventfd/eventfd01.c: Cannot open: Input/output error 2024-05-04T12:26:19.719 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/epoll_create1: Cannot mkdir: Input/output error 2024-05-04T12:26:19.719 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/epoll_create1/Makefile: Cannot open: Input/output error 2024-05-04T12:26:19.720 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/epoll_create1/epoll_create1_01.c: Cannot open: Input/output error 2024-05-04T12:26:19.720 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/signalfd4: Cannot mkdir: Input/output error 2024-05-04T12:26:19.720 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/signalfd4/Makefile: Cannot open: Input/output error 2024-05-04T12:26:19.720 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/signalfd4/signalfd4_01.c: Cannot open: Input/output error 2024-05-04T12:26:19.720 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/signalfd4/signalfd4_02.c: Cannot open: Input/output error 2024-05-04T12:26:19.720 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/setgroups: Cannot mkdir: Input/output error 2024-05-04T12:26:19.720 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/setgroups/setgroups04.c: Cannot open: Input/output error 2024-05-04T12:26:19.720 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/setgroups/compat_16.h: Cannot open: Input/output error 2024-05-04T12:26:19.720 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/setgroups/setgroups03.c: Cannot open: Input/output error 2024-05-04T12:26:19.721 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/setgroups/Makefile: Cannot open: Input/output error 2024-05-04T12:26:19.721 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/setgroups/setgroups01.c: Cannot open: Input/output error 2024-05-04T12:26:19.721 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/setgroups/setgroups02.c: Cannot open: Input/output error 2024-05-04T12:26:19.721 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/socketpair: Cannot mkdir: Input/output error 2024-05-04T12:26:19.721 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/socketpair/Makefile: Cannot open: Input/output error 2024-05-04T12:26:19.721 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/socketpair/socketpair01.c: Cannot open: Input/output error 2024-05-04T12:26:19.721 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/socketpair/socketpair02.c: Cannot open: Input/output error 2024-05-04T12:26:19.721 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/newuname: Cannot mkdir: Input/output error 2024-05-04T12:26:19.722 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/newuname/newuname01.c: Cannot open: Input/output error 2024-05-04T12:26:19.722 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/newuname/Makefile: Cannot open: Input/output error 2024-05-04T12:26:19.722 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/lstat: Cannot mkdir: Input/output error 2024-05-04T12:26:19.722 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/lstat/lstat01.c: Cannot open: Input/output error 2024-05-04T12:26:19.722 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/lstat/lstat02.c: Cannot open: Input/output error 2024-05-04T12:26:19.722 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/lstat/Makefile: Cannot open: Input/output error 2024-05-04T12:26:19.722 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/lstat/lstat03.c: Cannot open: Input/output error 2024-05-04T12:26:19.722 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/getdomainname: Cannot mkdir: Input/output error 2024-05-04T12:26:19.723 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/getdomainname/Makefile: Cannot open: Input/output error 2024-05-04T12:26:19.723 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/getdomainname/getdomainname01.c: Cannot open: Input/output error 2024-05-04T12:26:19.723 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/utime: Cannot mkdir: Input/output error 2024-05-04T12:26:19.723 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/utime/utime05.c: Cannot open: Input/output error 2024-05-04T12:26:19.723 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/utime/Makefile: Cannot open: Input/output error 2024-05-04T12:26:19.723 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/utime/utime04.c: Cannot open: Input/output error 2024-05-04T12:26:19.723 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/utime/utime06.c: Cannot open: Input/output error 2024-05-04T12:26:19.723 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/utime/utime01.c: Cannot open: Input/output error 2024-05-04T12:26:19.723 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/utime/utime03.c: Cannot open: Input/output error 2024-05-04T12:26:19.724 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/utime/utime02.c: Cannot open: Input/output error 2024-05-04T12:26:19.724 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/nice: Cannot mkdir: Input/output error 2024-05-04T12:26:19.724 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/nice/nice03.c: Cannot open: Input/output error 2024-05-04T12:26:19.724 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/nice/Makefile: Cannot open: Input/output error 2024-05-04T12:26:19.724 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/nice/nice01.c: Cannot open: Input/output error 2024-05-04T12:26:19.724 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/nice/nice05.c: Cannot open: Input/output error 2024-05-04T12:26:19.724 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/nice/nice02.c: Cannot open: Input/output error 2024-05-04T12:26:19.724 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/nice/nice04.c: Cannot open: Input/output error 2024-05-04T12:26:19.724 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/fchown: Cannot mkdir: Input/output error 2024-05-04T12:26:19.725 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/fchown/fchown04.c: Cannot open: Input/output error 2024-05-04T12:26:19.725 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/fchown/fchown03.c: Cannot open: Input/output error 2024-05-04T12:26:19.725 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/fchown/fchown05.c: Cannot open: Input/output error 2024-05-04T12:26:19.725 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/fchown/fchown02.c: Cannot open: Input/output error 2024-05-04T12:26:19.725 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/fchown/Makefile: Cannot open: Input/output error 2024-05-04T12:26:19.725 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/fchown/change_owner.mode.sh: Cannot open: Input/output error 2024-05-04T12:26:19.725 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/fchown/change_owner.c: Cannot open: Input/output error 2024-05-04T12:26:19.725 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/fchown/fchown01.c: Cannot open: Input/output error 2024-05-04T12:26:19.726 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/rename: Cannot mkdir: Input/output error 2024-05-04T12:26:19.726 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/rename/rename07.c: Cannot open: Input/output error 2024-05-04T12:26:19.726 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/rename/rename06.c: Cannot open: Input/output error 2024-05-04T12:26:19.726 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/rename/rename10.c: Cannot open: Input/output error 2024-05-04T12:26:19.726 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/rename/rename02.c: Cannot open: Input/output error 2024-05-04T12:26:19.726 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/rename/rename04.c: Cannot open: Input/output error 2024-05-04T12:26:19.726 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/rename/rename05.c: Cannot open: Input/output error 2024-05-04T12:26:19.726 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/rename/Makefile: Cannot open: Input/output error 2024-05-04T12:26:19.726 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/rename/rename12.c: Cannot open: Input/output error 2024-05-04T12:26:19.727 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/rename/rename09.c: Cannot open: Input/output error 2024-05-04T12:26:19.727 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/rename/rename08.c: Cannot open: Input/output error 2024-05-04T12:26:19.727 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/rename/rename13.c: Cannot open: Input/output error 2024-05-04T12:26:19.727 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/rename/rename03.c: Cannot open: Input/output error 2024-05-04T12:26:19.727 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/rename/rename01.c: Cannot open: Input/output error 2024-05-04T12:26:19.727 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/rename/rename14.c: Cannot open: Input/output error 2024-05-04T12:26:19.727 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/setegid: Cannot mkdir: Input/output error 2024-05-04T12:26:19.727 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/setegid/Makefile: Cannot open: Input/output error 2024-05-04T12:26:19.728 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/setegid/setegid01.c: Cannot open: Input/output error 2024-05-04T12:26:19.728 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/ppoll: Cannot mkdir: Input/output error 2024-05-04T12:26:19.728 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/ppoll/Makefile: Cannot open: Input/output error 2024-05-04T12:26:19.728 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/ppoll/ppoll01.c: Cannot open: Input/output error 2024-05-04T12:26:19.728 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/inotify: Cannot mkdir: Input/output error 2024-05-04T12:26:19.728 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/inotify/inotify02.c: Cannot open: Input/output error 2024-05-04T12:26:19.728 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/inotify/inotify01.c: Cannot open: Input/output error 2024-05-04T12:26:19.728 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/inotify/Makefile: Cannot open: Input/output error 2024-05-04T12:26:19.729 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/inotify/inotify03.c: Cannot open: Input/output error 2024-05-04T12:26:19.729 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/inotify/README: Cannot open: Input/output error 2024-05-04T12:26:19.729 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/modify_ldt: Cannot mkdir: Input/output error 2024-05-04T12:26:19.729 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/modify_ldt/modify_ldt01.c: Cannot open: Input/output error 2024-05-04T12:26:19.729 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/modify_ldt/Makefile: Cannot open: Input/output error 2024-05-04T12:26:19.729 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/modify_ldt/run-modify_ldt01.sh: Cannot open: Input/output error 2024-05-04T12:26:19.729 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/modify_ldt/modify_ldt02.c: Cannot open: Input/output error 2024-05-04T12:26:19.729 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/modify_ldt/run-modify_ldt02.sh: Cannot open: Input/output error 2024-05-04T12:26:19.729 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/setpgid: Cannot mkdir: Input/output error 2024-05-04T12:26:19.729 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/setpgid/Makefile: Cannot open: Input/output error 2024-05-04T12:26:19.730 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/setpgid/setpgid03.c: Cannot open: Input/output error 2024-05-04T12:26:19.730 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/setpgid/setpgid01.c: Cannot open: Input/output error 2024-05-04T12:26:19.730 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/setpgid/setpgid02.c: Cannot open: Input/output error 2024-05-04T12:26:19.730 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/timerfd: Cannot mkdir: Input/output error 2024-05-04T12:26:19.730 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/timerfd/timerfd02.c: Cannot open: Input/output error 2024-05-04T12:26:19.730 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/timerfd/Makefile: Cannot open: Input/output error 2024-05-04T12:26:19.730 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/timerfd/timerfd03.c: Cannot open: Input/output error 2024-05-04T12:26:19.730 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/timerfd/timerfd01.c: Cannot open: Input/output error 2024-05-04T12:26:19.731 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/accept4: Cannot mkdir: Input/output error 2024-05-04T12:26:19.731 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/accept4/accept4_01.c: Cannot open: Input/output error 2024-05-04T12:26:19.731 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/accept4/Makefile: Cannot open: Input/output error 2024-05-04T12:26:19.731 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/creat: Cannot mkdir: Input/output error 2024-05-04T12:26:19.731 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/creat/creat08.c: Cannot open: Input/output error 2024-05-04T12:26:19.731 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/creat/creat03.c: Cannot open: Input/output error 2024-05-04T12:26:19.731 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/creat/creat07.c: Cannot open: Input/output error 2024-05-04T12:26:19.731 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/creat/creat09.c: Cannot open: Input/output error 2024-05-04T12:26:19.731 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/creat/Makefile: Cannot open: Input/output error 2024-05-04T12:26:19.732 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/creat/creat04.c: Cannot open: Input/output error 2024-05-04T12:26:19.732 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/creat/creat06.c: Cannot open: Input/output error 2024-05-04T12:26:19.732 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/creat/creat01.c: Cannot open: Input/output error 2024-05-04T12:26:19.732 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/creat/creat05.c: Cannot open: Input/output error 2024-05-04T12:26:19.732 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/creat/test1.c: Cannot open: Input/output error 2024-05-04T12:26:19.732 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/sigtimedwait: Cannot mkdir: Input/output error 2024-05-04T12:26:19.732 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/sigtimedwait/Makefile: Cannot open: Input/output error 2024-05-04T12:26:19.732 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/send: Cannot mkdir: Input/output error 2024-05-04T12:26:19.733 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/send/send01.c: Cannot open: Input/output error 2024-05-04T12:26:19.733 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/send/Makefile: Cannot open: Input/output error 2024-05-04T12:26:19.733 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/chdir: Cannot mkdir: Input/output error 2024-05-04T12:26:19.733 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/chdir/chdir03.c: Cannot open: Input/output error 2024-05-04T12:26:19.733 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/chdir/Makefile: Cannot open: Input/output error 2024-05-04T12:26:19.733 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/chdir/chdir02.c: Cannot open: Input/output error 2024-05-04T12:26:19.733 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/chdir/chdir01.c: Cannot open: Input/output error 2024-05-04T12:26:19.733 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/chdir/chdir04.c: Cannot open: Input/output error 2024-05-04T12:26:19.733 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/munlock: Cannot mkdir: Input/output error 2024-05-04T12:26:19.734 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/munlock/Makefile: Cannot open: Input/output error 2024-05-04T12:26:19.734 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/munlock/munlock01.c: Cannot open: Input/output error 2024-05-04T12:26:19.734 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/munlock/munlock02.c: Cannot open: Input/output error 2024-05-04T12:26:19.734 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/asyncio: Cannot mkdir: Input/output error 2024-05-04T12:26:19.734 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/asyncio/Makefile: Cannot open: Input/output error 2024-05-04T12:26:19.734 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/asyncio/asyncio02.c: Cannot open: Input/output error 2024-05-04T12:26:19.734 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/wait4: Cannot mkdir: Input/output error 2024-05-04T12:26:19.734 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/wait4/Makefile: Cannot open: Input/output error 2024-05-04T12:26:19.735 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/wait4/wait402.c: Cannot open: Input/output error 2024-05-04T12:26:19.735 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/wait4/wait401.c: Cannot open: Input/output error 2024-05-04T12:26:19.735 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/munmap: Cannot mkdir: Input/output error 2024-05-04T12:26:19.735 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/munmap/munmap03.c: Cannot open: Input/output error 2024-05-04T12:26:19.735 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/munmap/Makefile: Cannot open: Input/output error 2024-05-04T12:26:19.735 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/munmap/munmap02.c: Cannot open: Input/output error 2024-05-04T12:26:19.735 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/munmap/munmap01.c: Cannot open: Input/output error 2024-05-04T12:26:19.735 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/signalfd: Cannot mkdir: Input/output error 2024-05-04T12:26:19.735 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/signalfd/Makefile: Cannot open: Input/output error 2024-05-04T12:26:19.736 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/signalfd/signalfd01.c: Cannot open: Input/output error 2024-05-04T12:26:19.736 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/mq_timedreceive: Cannot mkdir: Input/output error 2024-05-04T12:26:19.736 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/mq_timedreceive/Makefile: Cannot open: Input/output error 2024-05-04T12:26:19.736 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/mq_timedreceive/mq_timedreceive01.c: Cannot open: Input/output error 2024-05-04T12:26:19.736 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/symlinkat: Cannot mkdir: Input/output error 2024-05-04T12:26:19.736 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/symlinkat/Makefile: Cannot open: Input/output error 2024-05-04T12:26:19.736 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/symlinkat/symlinkat01.c: Cannot open: Input/output error 2024-05-04T12:26:19.736 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/statfs: Cannot mkdir: Input/output error 2024-05-04T12:26:19.737 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/statfs/statfs02.c: Cannot open: Input/output error 2024-05-04T12:26:19.737 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/statfs/statfs01.c: Cannot open: Input/output error 2024-05-04T12:26:19.737 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/statfs/Makefile: Cannot open: Input/output error 2024-05-04T12:26:19.737 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/statfs/statfs03.c: Cannot open: Input/output error 2024-05-04T12:26:19.737 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/lchown: Cannot mkdir: Input/output error 2024-05-04T12:26:19.737 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/lchown/create_link.c: Cannot open: Input/output error 2024-05-04T12:26:19.737 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/lchown/lchown02.c: Cannot open: Input/output error 2024-05-04T12:26:19.737 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/lchown/Makefile: Cannot open: Input/output error 2024-05-04T12:26:19.737 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/lchown/lchown01.c: Cannot open: Input/output error 2024-05-04T12:26:19.738 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/ptrace: Cannot mkdir: Input/output error 2024-05-04T12:26:19.738 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/ptrace/ptrace02.c: Cannot open: Input/output error 2024-05-04T12:26:19.738 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/ptrace/make_syscall_list.sh: Cannot open: Input/output error 2024-05-04T12:26:19.738 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/ptrace/spawn_ptrace_child.h: Cannot open: Input/output error 2024-05-04T12:26:19.738 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/ptrace/ptrace04.c: Cannot open: Input/output error 2024-05-04T12:26:19.738 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/ptrace/Makefile: Cannot open: Input/output error 2024-05-04T12:26:19.738 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/ptrace/ptrace.h: Cannot open: Input/output error 2024-05-04T12:26:19.738 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/ptrace/syscalls.h: Cannot open: Input/output error 2024-05-04T12:26:19.739 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/ptrace/ptrace01.c: Cannot open: Input/output error 2024-05-04T12:26:19.739 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/ptrace/ptrace03.c: Cannot open: Input/output error 2024-05-04T12:26:19.739 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/ptrace/ptrace06.c: Cannot open: Input/output error 2024-05-04T12:26:19.739 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/ptrace/ptrace05.c: Cannot open: Input/output error 2024-05-04T12:26:19.739 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/ptrace/simple_tracer.c: Cannot open: Input/output error 2024-05-04T12:26:19.739 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/sched_setscheduler: Cannot mkdir: Input/output error 2024-05-04T12:26:19.739 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/sched_setscheduler/Makefile: Cannot open: Input/output error 2024-05-04T12:26:19.739 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/sched_setscheduler/sched_setscheduler02.c: Cannot open: Input/output error 2024-05-04T12:26:19.739 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/sched_setscheduler/sched_setscheduler01.c: Cannot open: Input/output error 2024-05-04T12:26:19.740 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/setitimer: Cannot mkdir: Input/output error 2024-05-04T12:26:19.740 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/setitimer/setitimer02.c: Cannot open: Input/output error 2024-05-04T12:26:19.740 INFO:tasks.workunit.client.0.smithi079.stderr: 2024-05-04T12:26:19.740 INFO:tasks.workunit.client.0.smithi079.stderr:gzip: tar: ltp-full-20091231/testcases/kernel/syscalls/setitimer/Makefile: Cannot open: Input/output error 2024-05-04T12:26:19.740 INFO:tasks.workunit.client.0.smithi079.stderr:stdin: Stale file handle 2024-05-04T12:26:19.740 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/setitimer/setitimer03.c: Cannot open: Input/output error 2024-05-04T12:26:19.740 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/setitimer/setitimer01.c: Cannot open: Input/output error 2024-05-04T12:26:19.740 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/utimes: Cannot mkdir: Input/output error 2024-05-04T12:26:19.741 INFO:tasks.workunit.client.0.smithi079.stderr:tar: Unexpected EOF in archive 2024-05-04T12:26:19.741 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases/kernel: Cannot utime: Input/output error 2024-05-04T12:26:19.741 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231/testcases: Cannot utime: Input/output error 2024-05-04T12:26:19.741 INFO:tasks.workunit.client.0.smithi079.stderr:tar: ltp-full-20091231: Cannot utime: Input/output error 2024-05-04T12:26:19.741 INFO:tasks.workunit.client.0.smithi079.stderr:tar: Error is not recoverable: exiting now 2024-05-04T12:26:19.741 INFO:tasks.workunit:Stopping ['suites/fsstress.sh'] on client.0... 2024-05-04T12:26:19.742 DEBUG:teuthology.orchestra.run.smithi079:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-05-04T12:26:19.785 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-05-04T12:26:19.786 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-04T12:26:20.168 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:19.906+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:20.168 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:19.906+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:20.168 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:19.906+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:20.169 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:19.907+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:26:20.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:20.363+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:20.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:20.363+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:20.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:20.363+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:20.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:20.364+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:26:20.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:20.167+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:20.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:20.167+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:20.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:20.167+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:20.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:20.168+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:26:21.153 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:26:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:26:21.140+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 5935 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:26:21.154 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:20.901+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:21.154 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:20.901+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:21.154 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:20.901+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:21.154 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:20.903+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:26:21.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:21.182+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:21.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:21.182+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:21.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:21.182+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:21.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:21.183+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:26:21.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:21.357+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:21.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:21.357+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:21.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:21.357+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:21.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:21.358+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:26:22.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:22.146+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:22.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:22.146+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:22.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:22.146+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:22.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:22.147+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:26:22.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:21.927+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:22.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:21.927+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:22.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:21.927+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:22.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:21.928+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:26:22.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:22.401+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:22.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:22.401+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:22.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:22.401+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:22.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:22.402+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:26:23.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:23.172+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:23.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:23.172+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:23.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:23.172+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:23.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:23.173+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:26:23.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:22.928+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:23.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:22.928+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:23.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:22.928+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:23.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:22.929+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:26:23.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:23.393+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:23.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:23.393+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:23.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:23.393+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:23.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:23.394+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:26:24.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:23.934+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:24.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:23.934+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:24.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:23.934+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:24.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:23.935+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:26:24.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:24.394+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:24.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:24.394+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:24.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:24.394+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:24.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:24.395+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:26:24.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:24.206+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:24.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:24.207+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:24.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:24.207+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:24.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:24.208+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:26:25.370 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:24.981+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:25.371 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:24.981+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:25.371 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:24.981+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:25.371 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:24.981+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:26:25.371 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:25.195+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:25.371 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:25.195+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:25.372 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:25.195+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:25.372 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:25.196+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:26:25.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:25.369+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:25.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:25.369+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:25.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:25.369+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:25.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:25.370+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:26:26.267 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:26.007+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:26.268 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:26.007+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:26.268 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:26.007+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:26.268 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:26.008+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:26:26.268 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:26:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:26:26.141+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 6032 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:26:26.269 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:26.236+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:26.269 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:26.236+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:26.269 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:26.236+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:26.269 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:26.237+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:26:26.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:26.322+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:26.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:26.322+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:26.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:26.322+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:26.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:26.323+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:26:27.279 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:26.983+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:27.279 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:26.983+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:27.279 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:26.983+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:27.280 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:26.984+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:26:27.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:27.278+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:27.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:27.278+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:27.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:27.278+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:27.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:27.279+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:26:27.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:27.299+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:27.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:27.299+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:27.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:27.299+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:27.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:27.299+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:26:28.235 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:27.969+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:28.235 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:27.969+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:28.235 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:27.969+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:28.235 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:27.970+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:26:28.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:28.280+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:28.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:28.280+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:28.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:28.280+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:28.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:28.280+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:26:28.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:28.234+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:28.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:28.234+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:28.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:28.234+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:28.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:28.235+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:26:29.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:28.942+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:29.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:28.942+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:29.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:28.942+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:29.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:28.943+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:26:29.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:29.226+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:29.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:29.226+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:29.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:29.226+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:29.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:29.227+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:26:29.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:29.287+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:29.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:29.287+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:29.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:29.287+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:29.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:29.288+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:26:30.184 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:29.936+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:30.185 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:29.936+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:30.185 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:29.936+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:30.185 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:29.937+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:26:30.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:30.183+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:30.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:30.183+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:30.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:30.183+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:30.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:30.184+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:26:30.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:30.277+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:30.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:30.277+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:30.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:30.277+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:30.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:30.277+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:26:31.267 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:30.965+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:31.267 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:30.965+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:31.267 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:30.965+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:31.268 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:30.965+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:26:31.268 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:26:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:26:31.142+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 6125 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:26:31.268 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:31.173+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:31.268 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:31.173+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:31.268 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:31.173+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:31.269 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:31.174+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:26:31.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:31.299+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:31.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:31.299+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:31.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:31.299+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:31.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:31.300+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:26:32.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:32.134+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:32.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:32.134+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:32.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:32.134+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:32.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:32.135+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:26:32.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:31.924+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:32.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:31.924+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:32.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:31.924+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:32.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:31.925+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:26:32.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:32.323+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:32.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:32.323+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:32.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:32.323+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:32.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:32.324+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:26:33.336 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:33.179+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:33.336 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:33.179+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:33.336 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:33.179+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:33.336 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:33.180+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:26:33.337 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:32.962+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:33.337 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:32.962+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:33.337 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:32.962+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:33.337 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:32.962+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:26:33.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:33.335+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:33.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:33.335+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:33.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:33.335+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:33.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:33.336+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:26:34.322 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:34.150+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:34.323 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:34.150+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:34.323 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:34.150+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:34.323 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:34.151+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:26:34.323 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:34.005+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:34.324 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:34.005+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:34.324 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:34.005+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:34.324 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:34.005+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:26:34.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:34.321+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:34.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:34.321+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:34.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:34.321+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:34.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:34.322+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:26:35.293 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:34.972+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:35.293 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:34.972+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:35.293 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:34.972+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:35.293 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:34.972+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:26:35.294 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:35.190+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:35.294 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:35.190+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:35.294 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:35.190+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:35.294 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:35.191+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:26:35.696 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:35.292+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:35.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:35.292+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:35.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:35.292+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:35.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:35.293+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:26:36.195 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:26:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:26:36.143+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 6221 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:26:36.196 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:35.934+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:36.196 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:35.934+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:36.196 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:35.934+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:36.196 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:35.934+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:26:36.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:36.194+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:36.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:36.194+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:36.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:36.194+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:36.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:36.195+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:26:36.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:36.323+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:36.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:36.323+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:36.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:36.323+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:36.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:36.324+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:26:37.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:36.942+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:37.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:36.942+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:37.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:36.942+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:37.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:36.942+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:26:37.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:37.317+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:37.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:37.317+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:37.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:37.317+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:37.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:37.318+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:26:37.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:37.217+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:37.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:37.217+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:37.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:37.217+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:37.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:37.218+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:26:38.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:37.922+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:38.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:37.922+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:38.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:37.922+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:38.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:37.923+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:26:38.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:38.238+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:38.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:38.238+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:38.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:38.238+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:38.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:38.239+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:26:38.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:38.295+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:38.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:38.295+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:38.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:38.295+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:38.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:38.296+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:26:39.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:38.893+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:39.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:38.893+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:39.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:38.893+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:39.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:38.893+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:26:39.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:39.230+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:39.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:39.230+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:39.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:39.230+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:39.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:39.231+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:26:39.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:39.311+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:39.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:39.311+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:39.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:39.311+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:39.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:39.312+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:26:40.166 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:39.848+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:40.166 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:39.848+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:40.166 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:39.848+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:40.167 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:39.849+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:26:40.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:40.256+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:40.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:40.257+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:40.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:40.257+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:40.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:40.257+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:26:40.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:40.298+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:40.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:40.298+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:40.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:40.298+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:40.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:40.299+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:26:41.145 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:40.811+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:41.145 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:40.811+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:41.146 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:40.811+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:41.146 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:40.812+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:26:41.447 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:26:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:26:41.144+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 6315 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:26:41.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:41.262+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:41.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:41.262+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:41.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:41.262+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:41.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:41.263+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:26:41.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:41.296+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:41.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:41.296+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:41.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:41.296+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:41.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:41.297+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:26:42.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:41.784+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:42.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:41.784+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:42.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:41.784+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:42.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:41.785+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:26:42.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:42.308+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:42.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:42.308+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:42.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:42.308+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:42.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:42.309+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:26:42.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:42.258+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:42.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:42.258+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:42.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:42.258+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:42.699 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:42.259+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:26:43.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:42.815+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:43.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:42.815+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:43.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:42.815+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:43.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:42.816+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:26:43.544 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:43.297+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:43.545 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:43.297+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:43.545 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:43.297+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:43.545 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:43.298+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:26:43.545 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:43.286+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:43.545 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:43.286+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:43.545 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:43.286+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:43.546 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:43.287+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:26:44.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:43.848+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:44.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:43.848+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:44.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:43.848+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:44.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:43.849+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:26:44.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:44.280+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:44.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:44.280+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:44.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:44.280+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:44.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:44.281+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:26:44.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:44.328+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:44.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:44.328+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:44.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:44.328+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:44.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:44.328+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:26:45.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:44.803+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:45.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:44.803+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:45.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:44.803+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:45.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:44.804+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:26:45.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:45.308+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:45.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:45.308+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:45.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:45.308+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:45.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:45.309+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:26:45.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:45.318+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:45.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:45.318+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:45.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:45.318+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:45.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:45.319+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:26:46.146 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:45.770+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:46.146 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:45.770+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:46.147 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:45.770+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:46.147 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:45.771+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:26:46.447 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:26:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:26:46.145+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 6412 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:26:46.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:46.267+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:46.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:46.267+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:46.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:46.267+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:46.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:46.267+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:26:46.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:46.310+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:46.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:46.310+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:46.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:46.310+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:46.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:46.310+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:26:47.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:46.731+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:47.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:46.731+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:47.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:46.731+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:47.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:46.732+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:26:47.693 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:47.279+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:47.693 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:47.279+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:47.694 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:47.279+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:47.694 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:47.281+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:26:47.694 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:47.302+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:47.694 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:47.302+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:47.694 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:47.302+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:47.694 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:47.302+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:26:47.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:47.693+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:47.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:47.693+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:47.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:47.693+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:47.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:47.693+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:26:48.675 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:48.263+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:48.675 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:48.263+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:48.675 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:48.263+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:48.675 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:48.264+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:26:48.676 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:48.337+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:48.676 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:48.337+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:48.676 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:48.337+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:48.676 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:48.338+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:26:48.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:48.674+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:48.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:48.674+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:48.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:48.674+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:48.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:48.676+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:26:49.639 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:49.289+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:49.639 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:49.289+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:49.639 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:49.289+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:49.639 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:49.290+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:26:49.640 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:49.289+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:49.640 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:49.289+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:49.640 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:49.289+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:49.640 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:49.290+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:26:49.640 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:49.639+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:49.641 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:49.639+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:49.641 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:49.639+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:49.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:49.640+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:26:50.596 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:50.292+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:50.597 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:50.292+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:50.597 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:50.292+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:50.597 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:50.293+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:26:50.597 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:50.244+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:50.597 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:50.244+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:50.598 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:50.244+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:50.598 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:50.245+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:26:50.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:50.595+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:50.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:50.595+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:50.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:50.595+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:50.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:50.596+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:26:51.447 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:26:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:26:51.147+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 6506 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:26:51.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:51.315+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:51.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:51.315+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:51.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:51.315+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:51.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:51.316+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:26:51.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:51.254+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:51.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:51.254+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:51.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:51.254+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:51.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:51.255+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:26:51.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:51.581+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:51.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:51.581+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:51.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:51.581+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:51.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:51.582+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:26:52.623 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:52.311+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:52.623 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:52.311+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:52.624 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:52.311+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:52.624 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:52.311+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:26:52.624 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:52.244+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:52.624 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:52.244+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:52.624 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:52.244+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:52.624 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:52.245+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:26:52.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:52.622+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:52.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:52.622+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:52.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:52.622+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:52.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:52.623+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:26:53.545 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:53.348+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:53.545 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:53.348+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:53.545 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:53.348+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:53.545 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:53.349+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:26:53.545 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:53.239+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:53.546 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:53.239+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:53.546 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:53.239+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:53.546 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:53.240+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:26:53.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:53.670+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:53.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:53.670+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:53.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:53.670+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:53.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:53.671+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:26:54.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:54.324+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:54.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:54.324+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:54.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:54.324+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:54.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:54.325+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:26:54.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:54.244+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:54.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:54.244+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:54.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:54.244+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:54.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:54.245+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:26:55.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:54.718+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:55.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:54.718+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:55.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:54.718+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:55.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:54.720+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:26:55.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:55.328+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:55.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:55.328+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:55.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:55.328+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:55.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:55.328+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:26:55.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:55.261+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:55.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:55.261+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:55.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:55.261+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:55.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:55.262+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:26:56.148 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:55.718+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:56.149 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:55.718+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:56.149 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:55.718+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:56.149 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:55.718+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:26:56.446 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:26:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:26:56.147+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 6603 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:26:56.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:56.373+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:56.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:56.373+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:56.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:56.373+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:56.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:56.374+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:26:56.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:56.244+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:56.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:56.244+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:56.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:56.244+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:56.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:56.245+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:26:56.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:56.670+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:56.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:56.670+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:56.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:56.670+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:56.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:56.671+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:26:57.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:57.383+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:57.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:57.383+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:57.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:57.383+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:57.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:57.384+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:26:57.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:57.209+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:57.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:57.209+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:57.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:57.209+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:57.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:57.210+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:26:58.166 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:57.702+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:58.166 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:57.702+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:58.166 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:57.702+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:58.167 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:57.703+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:26:58.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:58.164+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:58.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:58.164+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:58.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:58.164+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:58.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:58.165+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:26:58.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:58.358+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:58.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:58.358+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:58.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:58.358+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:58.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:58.359+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:26:58.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:58.655+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:58.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:58.655+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:58.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:58.655+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:58.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:58.655+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:26:59.618 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:59.329+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:59.618 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:59.329+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:59.618 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:59.329+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:26:59.618 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:26:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:26:59.330+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:26:59.619 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:59.199+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:59.619 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:59.199+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:59.619 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:59.199+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:26:59.619 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:26:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:26:59.200+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:26:59.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:59.616+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:59.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:59.616+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:59.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:59.616+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:26:59.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:26:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:26:59.617+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:27:00.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:00.162+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:27:00.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:00.162+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:27:00.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:00.162+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:27:00.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:00.162+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:27:00.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:00.377+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:27:00.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:00.377+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:27:00.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:00.377+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:27:00.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:00.378+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:27:00.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:00.664+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:27:00.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:00.664+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:27:00.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:00.664+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:27:00.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:00.666+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:27:01.447 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:27:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:27:01.148+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 6697 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:27:01.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:01.374+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:27:01.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:01.374+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:27:01.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:01.374+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:27:01.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:01.375+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:27:01.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:01.175+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:27:01.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:01.175+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:27:01.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:01.175+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:27:01.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:01.175+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:27:02.170 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:01.698+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:27:02.170 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:01.698+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:27:02.171 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:01.699+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:27:02.171 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:01.699+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:27:02.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:02.360+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:27:02.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:02.360+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:27:02.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:02.360+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:27:02.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:02.361+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:27:02.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:02.169+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:27:02.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:02.169+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:27:02.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:02.169+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:27:02.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:02.170+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:27:02.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:02.675+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:27:02.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:02.675+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:27:02.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:02.675+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:27:02.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:02.676+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:27:03.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:03.312+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:27:03.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:03.312+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:27:03.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:03.312+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:27:03.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:03.313+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:27:03.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:03.150+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:27:03.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:03.150+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:27:03.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:03.150+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:27:03.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:03.151+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:27:03.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:03.678+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:27:03.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:03.678+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:27:03.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:03.678+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:27:03.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:03.679+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:27:04.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:04.358+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:27:04.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:04.358+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:27:04.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:04.358+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:27:04.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:04.359+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:27:04.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:04.155+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:27:04.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:04.155+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:27:04.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:04.155+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:27:04.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:04.156+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:27:05.115 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:04.705+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:27:05.115 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:04.705+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:27:05.116 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:04.705+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:27:05.116 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:04.706+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:27:05.399 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:05.114+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:27:05.399 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:05.114+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:27:05.399 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:05.114+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:27:05.399 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:05.115+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:27:05.660 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:05.398+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:27:05.660 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:05.398+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:27:05.661 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:05.398+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:27:05.661 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:05.398+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:27:05.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:05.659+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:27:05.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:05.659+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:27:05.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:05.659+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:27:05.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:05.660+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:27:06.402 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:27:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:27:06.149+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 6794 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:27:06.403 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:06.122+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:27:06.403 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:06.122+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:27:06.403 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:06.122+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:27:06.403 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:06.123+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:27:06.665 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:06.401+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:27:06.665 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:06.401+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:27:06.665 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:06.401+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:27:06.665 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:06.402+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:27:06.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:06.664+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:27:06.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:06.664+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:27:06.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:06.664+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:27:06.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:06.665+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:27:07.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:07.403+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:27:07.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:07.403+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:27:07.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:07.403+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:27:07.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:07.404+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:27:07.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:07.155+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:27:07.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:07.155+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:27:07.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:07.155+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:27:07.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:07.156+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:27:07.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:07.638+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:27:07.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:07.638+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:27:07.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:07.638+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:27:07.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:07.639+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:27:08.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:08.396+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:27:08.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:08.396+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:27:08.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:08.396+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:27:08.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:08.397+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:27:08.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:08.166+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:27:08.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:08.166+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:27:08.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:08.166+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:27:08.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:08.167+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:27:08.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:08.618+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:27:08.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:08.618+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:27:08.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:08.618+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:27:08.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:08.619+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:27:09.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:09.424+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:27:09.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:09.424+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:27:09.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:09.424+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:27:09.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:09.425+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:27:09.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:09.179+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:27:09.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:09.179+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:27:09.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:09.179+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:27:09.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:09.180+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:27:09.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:09.620+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:27:09.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:09.620+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:27:09.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:09.620+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:27:09.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:09.621+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:27:10.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:10.394+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:27:10.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:10.394+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:27:10.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:10.394+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:27:10.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:10.395+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:27:10.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:10.147+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:27:10.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:10.147+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:27:10.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:10.147+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:27:10.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:10.148+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:27:10.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:10.630+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:27:10.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:10.630+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:27:10.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:10.630+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:27:10.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:10.631+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:27:11.412 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:27:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:27:11.150+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 6888 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:27:11.412 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:11.152+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:27:11.412 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:11.152+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:27:11.412 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:11.152+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:27:11.413 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:11.153+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:27:11.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:11.411+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:27:11.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:11.411+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:27:11.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:11.411+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:27:11.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:11.412+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:27:11.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:11.626+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:27:11.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:11.626+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:27:11.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:11.626+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:27:11.699 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:11.627+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:27:12.414 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:12.134+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:27:12.415 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:12.134+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:27:12.415 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:12.134+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:27:12.415 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:12.134+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:27:12.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:12.580+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:27:12.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:12.580+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:27:12.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:12.580+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:27:12.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:12.581+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:27:12.699 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:12.413+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:27:12.699 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:12.413+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:27:12.699 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:12.413+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:27:12.699 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:12.414+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:27:13.397 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:13.094+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:27:13.397 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:13.094+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:27:13.397 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:13.094+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:27:13.397 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:13.095+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:27:13.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:13.565+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:27:13.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:13.565+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:27:13.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:13.565+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:27:13.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:13.566+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:27:13.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:13.396+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:27:13.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:13.396+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:27:13.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:13.396+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:27:13.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:13.396+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:27:14.413 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:14.122+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:27:14.413 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:14.122+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:27:14.414 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:14.122+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:27:14.414 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:14.123+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:27:14.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:14.412+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:27:14.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:14.412+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:27:14.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:14.412+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:27:14.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:14.413+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:27:14.699 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:14.575+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:27:14.699 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:14.575+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:27:14.699 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:14.575+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:27:14.699 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:14.576+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:27:15.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:15.155+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:27:15.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:15.155+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:27:15.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:15.155+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:27:15.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:15.155+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:27:15.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:15.549+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:27:15.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:15.549+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:27:15.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:15.549+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:27:15.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:15.550+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:27:15.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:15.454+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:27:15.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:15.454+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:27:15.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:15.454+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:27:15.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:15.455+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:27:16.415 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:27:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:27:16.151+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 6985 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:27:16.416 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:16.196+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:27:16.416 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:16.196+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:27:16.416 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:16.196+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:27:16.416 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:16.197+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:27:16.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:16.414+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:27:16.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:16.414+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:27:16.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:16.414+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:27:16.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:16.415+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:27:16.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:16.553+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:27:16.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:16.553+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:27:16.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:16.553+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:27:16.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:16.554+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:27:17.538 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:17.405+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:27:17.539 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:17.405+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:27:17.539 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:17.405+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:27:17.539 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:17.406+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:27:17.539 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:17.223+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:27:17.539 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:17.223+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:27:17.540 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:17.223+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:27:17.540 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:17.224+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:27:17.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:17.536+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:27:17.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:17.536+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:27:17.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:17.536+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:27:17.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:17.538+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:27:18.508 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:18.208+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:27:18.508 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:18.208+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:27:18.508 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:18.208+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:27:18.508 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:18.208+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:27:18.509 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:18.443+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:27:18.509 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:18.443+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:27:18.509 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:18.443+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:27:18.509 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:18.444+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:27:18.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:18.506+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.100305+0000 front 2024-05-04T12:16:49.100258+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:27:18.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:18.507+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.100247+0000 front 2024-05-04T12:16:49.100272+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:27:18.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:18.507+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.100233+0000 front 2024-05-04T12:16:49.100285+0000 (oldest deadline 2024-05-04T12:17:14.999885+0000) 2024-05-04T12:27:18.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:18.508+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:27:19.502 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:19.400+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:27:19.502 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:19.400+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:27:19.503 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:19.400+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:27:19.503 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:19.400+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:27:19.503 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:19.205+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:27:19.504 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:19.205+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:27:19.504 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:19.205+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:27:19.504 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:19.205+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:27:19.832 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:19.500+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:27:20.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:20.419+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:27:20.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:20.419+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:27:20.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:20.419+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:27:20.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:20.420+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:27:20.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:20.183+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:27:20.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:20.183+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:27:20.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:20.183+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:27:20.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:20.184+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:27:20.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:20.529+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:27:21.447 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:27:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:27:21.152+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 7079 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:27:21.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:21.377+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:27:21.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:21.377+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:27:21.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:21.377+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:27:21.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:21.377+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:27:21.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:21.137+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:27:21.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:21.137+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:27:21.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:21.137+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:27:21.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:21.138+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:27:21.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:21.566+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:27:22.405 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:22.111+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:27:22.406 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:22.111+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:27:22.406 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:22.112+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:27:22.406 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:22.112+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:27:22.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:22.568+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:27:22.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:22.404+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:27:22.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:22.404+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:27:22.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:22.404+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:27:22.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:22.405+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:27:23.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:23.365+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:27:23.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:23.365+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:27:23.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:23.365+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:27:23.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:23.366+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:27:23.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:23.116+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:27:23.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:23.116+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:27:23.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:23.116+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:27:23.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:23.117+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:27:23.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:23.605+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:27:24.398 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:24.073+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:49.297725+0000 front 2024-05-04T12:16:49.297743+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:27:24.399 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:24.073+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:49.297593+0000 front 2024-05-04T12:16:49.297690+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:27:24.399 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:24.073+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:49.297710+0000 front 2024-05-04T12:16:49.297664+0000 (oldest deadline 2024-05-04T12:17:15.197420+0000) 2024-05-04T12:27:24.399 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:24.074+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:27:24.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:24.645+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:27:24.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:24.397+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:27:24.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:24.397+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:27:24.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:24.397+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:27:24.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:24.398+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:27:25.353 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:25.075+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:27:25.652 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:25.352+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:27:25.652 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:25.352+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:27:25.652 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:25.352+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:27:25.652 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:25.353+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:27:25.652 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:25.597+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:27:26.380 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:26.114+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:27:26.381 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:27:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:27:26.153+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 7176 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:27:26.696 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:26.599+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:27:26.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:26.379+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:27:26.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:26.379+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:27:26.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:26.379+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:27:26.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:26.380+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:27:27.412 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:27.068+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:27:27.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:27.576+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:27:27.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:27.411+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:27:27.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:27.411+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:27:27.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:27.411+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:27:27.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:27.412+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:27:28.440 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:28.062+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:27:28.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:28.591+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:27:28.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:28.439+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:27:28.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:28.439+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:27:28.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:28.439+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:27:28.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:28.440+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:27:29.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:29.104+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:27:29.832 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:29.566+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:27:29.833 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:29.456+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:27:29.833 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:29.456+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:27:29.833 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:29.456+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:27:29.833 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:29.458+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:27:30.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:30.116+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:27:30.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:30.556+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:27:30.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:30.458+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:27:30.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:30.458+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:27:30.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:30.458+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:27:30.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:30.459+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:27:31.446 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:27:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:27:31.154+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 7270 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:27:31.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:31.108+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:27:31.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:31.568+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:27:31.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:31.457+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 since back 2024-05-04T12:16:50.907726+0000 front 2024-05-04T12:16:50.907700+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:27:31.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:31.457+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 since back 2024-05-04T12:16:50.907716+0000 front 2024-05-04T12:16:50.907756+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:27:31.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:31.457+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 since back 2024-05-04T12:16:50.907687+0000 front 2024-05-04T12:16:50.907675+0000 (oldest deadline 2024-05-04T12:17:15.607603+0000) 2024-05-04T12:27:31.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:31.458+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:27:32.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:32.157+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:27:32.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:32.455+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:27:32.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:32.579+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:27:33.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:33.147+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:27:33.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:33.532+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:27:33.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:33.492+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:27:34.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:34.132+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:27:34.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:34.483+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:27:34.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:34.515+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:27:35.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:35.149+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:27:35.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:35.533+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:27:35.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:35.512+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:27:36.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:36.159+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:27:36.447 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:27:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:27:36.156+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 7367 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:27:36.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:36.534+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:27:36.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:36.552+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:27:37.536 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:37.202+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:27:37.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:37.565+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:27:37.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:37.536+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:27:38.548 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:38.198+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:27:38.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:38.601+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:27:38.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:38.547+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:27:39.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:39.187+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:27:39.832 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:39.569+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:27:39.833 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:39.594+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:27:40.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:40.153+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:27:40.810 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:40.595+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:27:40.810 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:40.554+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:27:41.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:41.118+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:27:41.447 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:27:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:27:41.156+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 7460 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:27:41.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:41.595+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:27:41.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:41.557+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:27:42.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:42.127+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:27:42.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:42.556+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:27:42.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:42.607+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:27:42.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:42.607+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:27:42.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:42.607+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:27:42.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:42.608+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:27:43.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:43.142+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:27:43.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:43.592+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:27:43.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:43.634+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:27:43.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:43.634+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:27:43.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:43.634+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:27:43.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:43.635+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:27:44.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:44.157+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:27:44.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:44.635+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:27:44.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:44.585+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:27:44.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:44.585+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:27:44.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:44.585+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:27:44.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:44.587+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:27:45.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:45.136+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:27:45.860 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:45.612+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:27:45.861 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:45.546+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:27:45.861 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:45.546+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:27:45.861 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:45.546+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:27:45.861 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:45.547+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:27:46.447 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:27:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:27:46.157+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 7557 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:27:46.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:46.159+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:27:46.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:46.159+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:27:46.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:46.159+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:27:46.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:46.160+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:27:46.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:46.645+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:27:46.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:46.522+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:27:46.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:46.522+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:27:46.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:46.522+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:27:46.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:46.523+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:27:47.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:47.194+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:27:47.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:47.194+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:27:47.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:47.194+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:27:47.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:47.195+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:27:47.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:47.665+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:27:47.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:47.518+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:27:47.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:47.518+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:27:47.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:47.518+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:27:47.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:47.519+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:27:48.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:48.160+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:27:48.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:48.160+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:27:48.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:48.160+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:27:48.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:48.161+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:27:48.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:48.632+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:27:48.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:48.528+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:27:48.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:48.528+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:27:48.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:48.528+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:27:48.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:48.529+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:27:49.498 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:49.196+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:27:49.498 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:49.196+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:27:49.498 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:49.196+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:27:49.498 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:49.197+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:27:49.832 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:49.667+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:27:49.833 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:49.497+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:27:49.833 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:49.497+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:27:49.833 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:49.497+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:27:49.833 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:49.497+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:27:50.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:50.191+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:27:50.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:50.191+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:27:50.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:50.191+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:27:50.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:50.191+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:27:50.911 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:50.622+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:27:50.911 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:50.518+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:27:50.911 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:50.518+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:27:50.911 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:50.518+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:27:50.912 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:50.519+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:27:51.172 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:27:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:27:51.159+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 7651 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:27:51.172 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:51.143+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:27:51.172 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:51.143+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:27:51.172 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:51.143+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:27:51.172 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:51.145+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:27:51.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:51.627+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:27:51.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:51.503+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:27:51.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:51.503+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:27:51.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:51.503+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:27:51.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:51.504+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:27:52.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:52.139+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:27:52.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:52.139+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:27:52.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:52.139+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:27:52.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:52.140+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:27:52.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:52.596+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:27:52.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:52.504+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:27:52.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:52.504+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:27:52.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:52.504+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:27:52.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:52.505+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:27:53.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:53.123+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:27:53.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:53.123+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:27:53.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:53.123+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:27:53.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:53.124+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:27:53.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:53.610+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:27:53.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:53.519+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:27:53.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:53.519+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:27:53.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:53.519+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:27:53.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:53.520+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:27:54.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:54.132+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:27:54.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:54.132+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:27:54.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:54.132+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:27:54.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:54.132+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:27:54.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:54.575+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:27:54.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:54.478+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:27:54.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:54.478+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:27:54.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:54.478+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:27:54.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:54.479+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:27:55.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:55.086+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:27:55.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:55.086+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:27:55.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:55.086+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:27:55.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:55.087+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:27:55.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:55.567+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:27:55.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:55.567+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:27:55.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:55.567+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:27:55.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:55.567+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:27:55.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:55.457+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:27:55.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:55.457+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:27:55.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:55.457+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:27:55.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:55.457+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:27:56.284 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:27:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:27:56.160+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 7748 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:27:56.284 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:56.089+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:27:56.284 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:56.089+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:27:56.284 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:56.089+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:27:56.284 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:56.090+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:27:56.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:56.460+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:27:56.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:56.460+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:27:56.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:56.460+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:27:56.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:56.461+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:27:56.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:56.519+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:27:56.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:56.519+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:27:56.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:56.519+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:27:56.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:56.519+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:27:57.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:57.089+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:27:57.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:57.089+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:27:57.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:57.089+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:27:57.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:57.090+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:27:57.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:57.464+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:27:57.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:57.464+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:27:57.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:57.464+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:27:57.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:57.465+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:27:57.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:57.562+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:27:57.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:57.562+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:27:57.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:57.562+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:27:57.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:57.563+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:27:58.427 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:58.067+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:27:58.427 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:58.067+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:27:58.427 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:58.067+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:27:58.427 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:58.068+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:27:58.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:58.426+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:27:58.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:58.426+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:27:58.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:58.426+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:27:58.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:58.427+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:27:58.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:58.551+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:27:58.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:58.551+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:27:58.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:58.551+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:27:58.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:58.552+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:27:59.384 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:59.032+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:27:59.384 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:59.032+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:27:59.384 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:59.032+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:27:59.384 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:27:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:27:59.033+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:27:59.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:59.382+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:27:59.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:59.382+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:27:59.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:59.382+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:27:59.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:27:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:27:59.383+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:27:59.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:59.542+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:27:59.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:59.542+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:27:59.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:59.542+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:27:59.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:27:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:27:59.543+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:28:00.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:00.079+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:00.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:00.079+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:00.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:00.079+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:00.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:00.080+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:28:00.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:00.372+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:00.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:00.372+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:00.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:00.372+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:00.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:00.373+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:28:00.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:00.553+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:00.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:00.553+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:00.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:00.553+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:00.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:00.554+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:28:01.287 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:28:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:28:01.161+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 7843 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:28:01.287 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:01.127+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:01.288 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:01.127+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:01.288 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:01.127+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:01.288 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:01.128+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:28:01.543 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:01.355+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:01.543 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:01.355+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:01.544 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:01.355+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:01.544 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:01.355+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:28:01.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:01.542+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:01.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:01.542+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:01.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:01.542+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:01.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:01.543+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:28:02.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:02.116+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:02.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:02.116+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:02.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:02.116+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:02.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:02.117+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:28:02.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:02.307+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:02.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:02.307+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:02.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:02.307+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:02.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:02.307+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:28:02.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:02.513+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:02.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:02.513+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:02.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:02.513+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:02.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:02.514+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:28:03.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:03.349+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:03.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:03.349+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:03.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:03.349+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:03.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:03.350+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:28:03.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:03.139+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:03.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:03.139+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:03.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:03.139+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:03.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:03.139+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:28:03.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:03.480+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:03.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:03.480+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:03.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:03.480+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:03.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:03.480+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:28:04.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:04.370+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:04.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:04.370+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:04.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:04.370+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:04.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:04.371+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:28:04.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:04.128+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:04.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:04.128+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:04.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:04.128+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:04.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:04.129+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:28:04.731 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:04.447+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:04.732 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:04.447+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:04.732 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:04.447+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:04.732 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:04.447+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:28:05.407 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:05.087+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:05.407 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:05.087+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:05.407 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:05.087+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:05.407 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:05.088+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:28:05.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:05.441+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:05.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:05.441+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:05.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:05.441+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:05.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:05.442+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:28:05.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:05.405+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:05.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:05.405+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:05.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:05.405+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:05.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:05.406+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:28:06.447 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:28:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:28:06.163+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 7941 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:28:06.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:06.106+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:06.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:06.106+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:06.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:06.106+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:06.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:06.107+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:28:06.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:06.468+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:06.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:06.468+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:06.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:06.468+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:06.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:06.468+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:28:06.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:06.456+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:06.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:06.456+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:06.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:06.456+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:06.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:06.457+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:28:07.426 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:07.072+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:07.426 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:07.072+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:07.426 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:07.073+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:07.427 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:07.073+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:28:07.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:07.493+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:07.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:07.493+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:07.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:07.493+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:07.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:07.493+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:28:07.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:07.425+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:07.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:07.425+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:07.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:07.425+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:07.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:07.425+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:28:08.427 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:08.058+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:08.427 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:08.058+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:08.427 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:08.058+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:08.428 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:08.059+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:28:08.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:08.426+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:08.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:08.426+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:08.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:08.426+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:08.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:08.427+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:28:08.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:08.466+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:08.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:08.466+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:08.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:08.466+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:08.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:08.467+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:28:09.423 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:09.044+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:09.424 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:09.044+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:09.424 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:09.044+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:09.424 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:09.045+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:28:09.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:09.422+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:09.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:09.422+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:09.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:09.422+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:09.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:09.423+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:28:09.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:09.469+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:09.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:09.469+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:09.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:09.469+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:09.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:09.470+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:28:10.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:10.047+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:10.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:10.047+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:10.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:10.047+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:10.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:10.048+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:28:10.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:10.380+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:10.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:10.380+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:10.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:10.380+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:10.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:10.381+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:28:10.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:10.439+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:10.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:10.439+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:10.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:10.439+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:10.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:10.439+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:28:11.287 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:28:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:28:11.164+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 8036 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:28:11.287 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:11.034+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:11.287 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:11.034+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:11.287 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:11.034+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:11.287 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:11.035+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:28:11.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:11.388+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:11.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:11.388+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:11.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:11.388+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:11.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:11.389+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:28:11.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:11.489+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:11.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:11.489+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:11.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:11.489+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:11.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:11.489+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:28:12.348 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:12.040+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:12.348 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:12.040+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:12.348 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:12.040+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:12.348 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:12.041+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:28:12.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:12.346+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:12.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:12.346+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:12.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:12.346+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:12.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:12.347+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:28:12.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:12.534+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:12.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:12.534+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:12.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:12.534+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:12.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:12.535+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:28:13.383 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:13.024+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:13.383 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:13.024+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:13.383 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:13.024+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:13.384 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:13.025+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:28:13.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:13.554+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:13.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:13.554+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:13.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:13.554+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:13.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:13.555+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:28:13.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:13.382+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:13.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:13.382+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:13.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:13.382+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:13.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:13.382+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:28:14.369 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:14.051+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:14.369 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:14.051+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:14.369 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:14.051+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:14.369 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:14.052+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:28:14.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:14.510+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:14.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:14.510+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:14.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:14.510+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:14.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:14.511+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:28:14.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:14.367+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:14.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:14.368+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:14.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:14.368+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:14.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:14.369+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:28:15.337 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:15.069+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:15.337 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:15.069+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:15.337 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:15.069+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:15.338 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:15.070+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:28:15.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:15.336+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:15.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:15.336+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:15.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:15.336+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:15.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:15.337+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:28:15.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:15.525+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:15.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:15.525+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:15.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:15.525+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:15.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:15.526+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:28:16.355 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:28:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:28:16.165+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 8134 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:28:16.355 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:16.087+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:16.356 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:16.087+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:16.356 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:16.087+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:16.356 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:16.088+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:28:16.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:16.354+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:16.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:16.354+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:16.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:16.354+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:16.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:16.355+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:28:16.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:16.517+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:16.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:16.517+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:16.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:16.517+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:16.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:16.518+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:28:17.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:17.305+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:17.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:17.306+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:17.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:17.306+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:17.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:17.306+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:28:17.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:17.100+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:17.449 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:17.100+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:17.449 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:17.100+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:17.449 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:17.100+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:28:17.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:17.548+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:17.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:17.548+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:17.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:17.548+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:17.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:17.548+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:28:18.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:18.278+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:18.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:18.278+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:18.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:18.278+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:18.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:18.278+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:28:18.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:18.092+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:18.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:18.092+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:18.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:18.092+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:18.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:18.092+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:28:18.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:18.581+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:18.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:18.581+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:18.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:18.581+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:18.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:18.581+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:28:19.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:19.287+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:19.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:19.287+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:19.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:19.287+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:19.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:19.288+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:28:19.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:19.139+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:19.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:19.139+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:19.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:19.139+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:19.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:19.140+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:28:19.832 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:19.564+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:19.832 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:19.564+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:19.833 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:19.564+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:19.833 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:19.565+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:28:20.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:20.285+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:20.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:20.286+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:20.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:20.286+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:20.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:20.287+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:28:20.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:20.158+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:20.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:20.158+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:20.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:20.158+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:20.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:20.159+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:28:20.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:20.555+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:20.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:20.555+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:20.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:20.555+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:20.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:20.556+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:28:21.447 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:28:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:28:21.167+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 8229 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:28:21.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:21.125+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:21.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:21.125+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:21.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:21.125+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:21.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:21.126+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:28:21.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:21.272+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:21.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:21.272+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:21.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:21.272+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:21.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:21.273+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:28:21.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:21.536+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:21.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:21.536+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:21.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:21.536+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:21.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:21.537+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:28:22.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:22.314+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:22.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:22.314+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:22.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:22.314+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:22.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:22.315+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:28:22.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:22.120+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:22.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:22.120+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:22.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:22.120+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:22.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:22.120+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:28:22.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:22.516+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:22.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:22.516+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:22.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:22.516+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:22.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:22.517+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:28:23.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:23.352+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:23.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:23.352+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:23.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:23.352+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:23.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:23.353+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:28:23.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:23.120+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:23.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:23.120+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:23.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:23.120+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:23.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:23.121+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:28:23.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:23.542+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:23.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:23.542+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:23.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:23.542+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:23.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:23.542+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:28:24.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:24.314+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:24.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:24.314+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:24.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:24.314+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:24.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:24.314+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:28:24.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:24.157+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:24.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:24.157+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:24.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:24.157+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:24.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:24.158+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:28:24.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:24.522+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:24.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:24.522+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:24.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:24.522+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:24.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:24.522+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:28:25.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:25.345+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:25.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:25.345+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:25.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:25.345+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:25.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:25.346+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:28:25.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:25.158+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:25.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:25.158+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:25.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:25.158+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:25.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:25.159+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:28:25.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:25.512+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:25.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:25.512+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:25.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:25.512+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:25.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:25.513+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:28:26.447 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:28:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:28:26.167+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 8327 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:28:26.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:26.310+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:26.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:26.310+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:26.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:26.310+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:26.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:26.310+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:28:26.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:26.155+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:26.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:26.155+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:26.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:26.155+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:26.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:26.155+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:28:26.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:26.528+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:26.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:26.528+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:26.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:26.528+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:26.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:26.529+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:28:27.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:27.272+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:27.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:27.272+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:27.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:27.272+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:27.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:27.273+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:28:27.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:27.122+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:27.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:27.122+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:27.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:27.122+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:27.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:27.123+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:28:27.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:27.481+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:27.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:27.481+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:27.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:27.481+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:27.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:27.482+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:28:28.443 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:28.165+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:28.443 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:28.165+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:28.443 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:28.165+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:28.443 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:28.166+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:28:28.443 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:28.306+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:28.444 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:28.306+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:28.444 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:28.306+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:28.444 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:28.307+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:28:28.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:28.441+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:28.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:28.441+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:28.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:28.441+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:28.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:28.442+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:28:29.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:29.296+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:29.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:29.296+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:29.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:29.296+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:29.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:29.297+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:28:29.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:29.138+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:29.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:29.138+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:29.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:29.138+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:29.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:29.138+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:28:29.833 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:29.479+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:29.833 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:29.479+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:29.833 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:29.479+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:29.833 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:29.480+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:28:30.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:30.255+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:30.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:30.255+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:30.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:30.255+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:30.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:30.256+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:28:30.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:30.128+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:30.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:30.128+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:30.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:30.128+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:30.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:30.128+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:28:30.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:30.514+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:30.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:30.514+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:30.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:30.514+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:30.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:30.515+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:28:31.447 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:28:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:28:31.168+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 8422 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:28:31.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:31.217+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:31.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:31.217+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:31.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:31.217+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:31.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:31.218+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:28:31.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:31.107+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:31.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:31.107+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:31.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:31.107+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:31.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:31.108+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:28:31.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:31.506+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:31.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:31.506+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:31.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:31.506+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:31.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:31.507+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:28:32.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:32.176+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:32.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:32.176+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:32.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:32.176+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:32.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:32.177+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:28:32.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:32.151+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:32.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:32.151+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:32.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:32.151+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:32.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:32.152+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:28:32.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:32.549+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:32.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:32.549+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:32.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:32.549+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:32.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:32.550+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:28:33.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:33.213+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:33.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:33.213+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:33.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:33.213+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:33.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:33.214+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:28:33.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:33.108+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:33.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:33.108+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:33.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:33.108+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:33.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:33.109+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:28:33.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:33.594+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:33.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:33.594+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:33.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:33.594+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:33.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:33.595+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:28:34.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:34.243+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:34.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:34.243+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:34.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:34.243+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:34.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:34.244+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:28:34.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:34.139+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:34.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:34.139+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:34.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:34.139+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:34.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:34.140+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:28:34.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:34.611+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:34.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:34.611+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:34.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:34.611+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:34.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:34.612+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:28:35.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:35.247+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:35.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:35.247+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:35.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:35.247+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:35.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:35.248+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:28:35.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:35.167+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:35.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:35.167+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:35.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:35.167+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:35.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:35.168+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:28:35.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:35.636+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:35.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:35.636+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:35.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:35.636+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:35.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:35.636+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:28:36.447 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:28:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:28:36.169+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 8520 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:28:36.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:36.270+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:36.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:36.270+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:36.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:36.270+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:36.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:36.271+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:28:36.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:36.199+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:36.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:36.199+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:36.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:36.199+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:36.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:36.199+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:28:36.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:36.586+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:36.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:36.586+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:36.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:36.586+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:36.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:36.587+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:28:37.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:37.271+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:37.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:37.271+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:37.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:37.272+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:37.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:37.272+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:28:37.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:37.160+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:37.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:37.160+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:37.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:37.160+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:37.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:37.160+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:28:37.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:37.603+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:37.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:37.603+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:37.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:37.603+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:37.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:37.604+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:28:38.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:38.253+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:38.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:38.253+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:38.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:38.253+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:38.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:38.254+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:28:38.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:38.182+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:38.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:38.182+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:38.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:38.182+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:38.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:38.183+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:28:38.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:38.572+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:38.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:38.572+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:38.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:38.572+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:38.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:38.573+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:28:39.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:39.222+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:39.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:39.222+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:39.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:39.222+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:39.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:39.223+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:28:39.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:39.161+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:39.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:39.161+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:39.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:39.161+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:39.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:39.162+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:28:39.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:39.616+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:39.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:39.616+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:39.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:39.616+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:39.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:39.616+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:28:40.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:40.212+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:40.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:40.212+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:40.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:40.212+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:40.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:40.213+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:28:40.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:40.185+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:40.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:40.185+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:40.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:40.185+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:40.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:40.186+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:28:40.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:40.572+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:40.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:40.572+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:40.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:40.572+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:40.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:40.573+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:28:41.434 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:28:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:28:41.171+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 8615 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:28:41.435 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:41.217+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:41.435 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:41.217+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:41.435 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:41.217+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:41.435 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:41.217+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:28:41.435 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:41.185+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:41.436 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:41.185+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:41.436 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:41.185+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:41.436 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:41.186+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:28:41.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:41.525+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:41.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:41.525+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:41.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:41.525+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:41.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:41.526+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:28:42.510 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:42.219+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:42.511 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:42.219+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:42.511 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:42.219+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:42.511 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:42.219+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:28:42.511 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:42.249+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:42.511 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:42.249+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:42.512 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:42.249+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:42.512 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:42.250+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:28:42.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:42.509+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:42.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:42.509+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:42.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:42.509+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:42.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:42.510+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:28:43.553 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:43.258+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:43.553 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:43.258+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:43.554 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:43.258+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:43.554 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:43.259+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:28:43.554 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:43.230+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:43.554 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:43.230+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:43.554 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:43.230+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:43.554 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:43.231+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:28:43.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:43.554+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:43.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:43.554+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:43.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:43.554+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:43.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:43.555+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:28:44.536 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:44.220+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:44.537 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:44.220+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:44.537 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:44.220+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:44.537 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:44.221+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:28:44.537 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:44.279+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:44.537 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:44.279+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:44.538 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:44.279+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:44.538 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:44.279+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:28:44.923 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:44.535+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:44.923 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:44.535+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:44.923 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:44.535+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:44.923 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:44.536+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:28:45.535 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:45.209+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:45.536 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:45.209+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:45.536 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:45.209+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:45.536 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:45.210+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:28:45.536 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:45.297+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:45.536 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:45.297+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:45.536 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:45.297+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:45.537 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:45.298+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:28:45.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:45.534+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:45.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:45.534+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:45.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:45.534+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:45.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:45.535+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:28:46.447 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:28:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:28:46.172+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 8713 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:28:46.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:46.252+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:46.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:46.253+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:46.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:46.253+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:46.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:46.254+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:28:46.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:46.297+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:46.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:46.297+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:46.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:46.297+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:46.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:46.298+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:28:46.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:46.531+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:46.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:46.531+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:46.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:46.531+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:46.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:46.532+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:28:47.505 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:47.235+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:47.505 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:47.235+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:47.505 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:47.235+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:47.505 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:47.236+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:28:47.505 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:47.292+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:47.506 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:47.292+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:47.506 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:47.292+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:47.506 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:47.293+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:28:47.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:47.503+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:47.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:47.503+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:47.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:47.503+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:47.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:47.504+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:28:48.540 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:48.255+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:48.541 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:48.255+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:48.541 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:48.255+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:48.541 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:48.256+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:28:48.541 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:48.245+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:48.541 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:48.245+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:48.541 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:48.245+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:48.542 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:48.246+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:28:48.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:48.539+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:48.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:48.539+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:48.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:48.539+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:48.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:48.540+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:28:49.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:49.287+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:49.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:49.287+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:49.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:49.287+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:49.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:49.287+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:28:49.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:49.497+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:49.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:49.497+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:49.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:49.497+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:49.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:49.498+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:28:49.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:49.293+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:49.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:49.293+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:49.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:49.293+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:49.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:49.293+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:28:50.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:50.479+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:50.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:50.479+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:50.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:50.479+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:50.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:50.480+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:28:50.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:50.273+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:50.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:50.273+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:50.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:50.273+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:50.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:50.273+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:28:50.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:50.322+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:50.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:50.322+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:50.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:50.322+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:50.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:50.323+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:28:51.447 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:28:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:28:51.173+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 8808 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:28:51.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:51.319+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:51.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:51.319+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:51.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:51.319+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:51.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:51.320+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:28:51.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:51.332+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:51.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:51.332+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:51.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:51.332+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:51.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:51.333+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:28:51.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:51.499+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:51.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:51.499+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:51.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:51.499+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:51.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:51.500+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:28:52.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:52.453+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:52.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:52.453+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:52.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:52.453+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:52.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:52.453+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:28:52.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:52.322+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:52.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:52.322+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:52.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:52.322+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:52.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:52.323+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:28:52.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:52.362+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:52.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:52.362+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:52.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:52.362+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:52.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:52.363+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:28:53.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:53.339+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:53.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:53.339+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:53.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:53.339+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:53.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:53.339+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:28:53.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:53.489+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:53.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:53.489+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:53.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:53.489+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:53.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:53.490+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:28:53.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:53.345+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:53.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:53.345+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:53.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:53.345+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:53.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:53.346+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:28:54.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:54.294+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:54.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:54.294+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:54.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:54.294+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:54.446 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:54.295+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:28:54.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:54.364+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:54.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:54.364+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:54.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:54.364+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:54.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:54.365+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:28:54.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:54.445+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:54.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:54.445+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:54.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:54.445+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:54.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:54.445+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:28:55.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:55.320+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:55.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:55.320+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:55.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:55.321+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:55.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:55.321+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:28:55.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:55.407+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:55.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:55.407+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:55.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:55.407+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:55.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:55.408+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:28:55.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:55.327+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:55.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:55.327+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:55.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:55.327+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:55.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:55.328+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:28:56.444 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:56.322+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:56.444 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:56.322+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:56.444 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:56.322+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:56.444 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:56.323+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:28:56.444 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:56.341+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:56.445 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:56.341+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:56.445 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:56.341+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:56.445 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:56.343+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:28:56.445 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:28:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:28:56.174+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 8906 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:28:56.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:56.442+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:56.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:56.442+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:56.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:56.442+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:56.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:56.443+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:28:57.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:57.324+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:57.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:57.324+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:57.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:57.324+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:57.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:57.325+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:28:57.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:57.453+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:57.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:57.453+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:57.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:57.453+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:57.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:57.455+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:28:57.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:57.344+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:57.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:57.344+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:57.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:57.344+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:57.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:57.345+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:28:58.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:58.341+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:58.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:58.341+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:58.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:58.341+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:58.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:58.341+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:28:58.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:58.420+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:58.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:58.420+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:58.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:58.420+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:58.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:58.421+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:28:58.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:58.338+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:58.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:58.338+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:58.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:58.338+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:58.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:58.338+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:28:59.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:59.387+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:59.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:59.387+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:59.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:59.387+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:28:59.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:28:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:28:59.388+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:28:59.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:59.465+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:59.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:59.465+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:59.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:59.465+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:28:59.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:28:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:28:59.465+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:28:59.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:59.385+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:59.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:59.385+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:59.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:59.385+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:28:59.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:28:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:28:59.386+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:29:00.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:00.434+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:00.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:00.434+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:00.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:00.434+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:00.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:00.434+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:29:00.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:00.363+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:00.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:00.363+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:00.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:00.363+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:00.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:00.364+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:29:00.699 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:00.493+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:00.699 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:00.493+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:00.699 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:00.493+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:00.699 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:00.494+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:29:01.447 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:29:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:29:01.175+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 9001 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:29:01.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:01.397+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:01.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:01.397+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:01.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:01.397+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:01.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:01.398+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:29:01.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:01.396+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:01.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:01.396+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:01.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:01.396+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:01.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:01.396+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:29:01.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:01.499+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:01.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:01.499+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:01.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:01.499+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:01.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:01.500+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:29:02.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:02.366+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:02.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:02.366+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:02.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:02.366+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:02.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:02.367+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:29:02.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:02.539+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:02.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:02.539+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:02.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:02.539+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:02.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:02.541+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:29:02.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:02.384+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:02.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:02.384+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:02.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:02.384+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:02.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:02.385+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:29:03.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:03.343+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:03.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:03.343+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:03.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:03.343+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:03.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:03.344+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:29:03.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:03.507+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:03.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:03.507+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:03.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:03.507+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:03.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:03.508+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:29:03.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:03.370+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:03.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:03.370+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:03.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:03.370+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:03.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:03.371+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:29:04.598 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:04.380+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:04.598 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:04.380+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:04.599 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:04.380+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:04.599 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:04.380+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:29:04.599 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:04.508+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:04.599 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:04.508+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:04.599 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:04.508+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:04.599 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:04.509+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:29:04.600 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:04.304+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:04.600 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:04.304+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:04.600 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:04.304+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:04.600 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:04.304+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:29:05.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:05.313+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:05.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:05.313+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:05.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:05.313+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:05.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:05.314+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:29:05.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:05.393+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:05.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:05.393+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:05.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:05.393+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:05.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:05.394+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:29:05.699 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:05.499+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:05.699 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:05.499+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:05.699 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:05.499+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:05.699 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:05.500+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:29:06.428 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:29:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:29:06.177+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 9099 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:29:06.429 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:06.411+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:06.429 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:06.411+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:06.429 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:06.411+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:06.429 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:06.412+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:29:06.429 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:06.292+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:06.429 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:06.292+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:06.429 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:06.292+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:06.430 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:06.293+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:29:06.694 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:06.526+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:06.694 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:06.526+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:06.695 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:06.526+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:06.695 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:06.527+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:29:07.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:07.341+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:07.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:07.341+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:07.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:07.341+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:07.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:07.342+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:29:07.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:07.508+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:07.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:07.508+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:07.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:07.508+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:07.699 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:07.508+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:29:07.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:07.440+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:07.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:07.440+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:07.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:07.440+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:07.700 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:07.440+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:29:08.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:08.315+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:08.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:08.315+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:08.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:08.315+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:08.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:08.316+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:29:08.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:08.522+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:08.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:08.522+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:08.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:08.522+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:08.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:08.524+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:29:08.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:08.421+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:08.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:08.421+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:08.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:08.421+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:08.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:08.422+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:29:09.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:09.364+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:09.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:09.364+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:09.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:09.364+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:09.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:09.365+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:29:09.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:09.507+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:09.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:09.507+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:09.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:09.507+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:09.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:09.508+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:29:09.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:09.378+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:09.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:09.378+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:09.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:09.378+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:09.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:09.379+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:29:10.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:10.403+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:10.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:10.403+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:10.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:10.403+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:10.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:10.405+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:29:10.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:10.506+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:10.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:10.506+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:10.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:10.506+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:10.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:10.507+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:29:10.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:10.421+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:10.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:10.421+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:10.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:10.421+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:10.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:10.422+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:29:11.447 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:29:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:29:11.177+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 9193 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:29:11.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:11.394+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:11.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:11.394+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:11.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:11.394+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:11.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:11.395+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:29:11.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:11.413+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:11.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:11.413+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:11.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:11.413+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:11.449 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:11.414+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:29:11.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:11.514+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:11.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:11.514+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:11.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:11.514+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:11.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:11.515+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:29:12.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:12.412+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:12.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:12.412+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:12.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:12.412+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:12.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:12.413+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:29:12.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:12.474+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:12.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:12.474+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:12.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:12.474+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:12.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:12.475+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:29:12.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:12.412+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:12.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:12.412+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:12.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:12.412+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:12.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:12.413+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:29:13.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:13.449+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:13.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:13.449+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:13.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:13.449+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:13.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:13.450+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:29:13.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:13.492+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:13.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:13.492+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:13.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:13.492+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:13.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:13.493+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:29:13.699 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:13.430+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:13.699 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:13.430+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:13.699 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:13.430+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:13.699 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:13.431+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:29:14.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:14.457+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:14.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:14.457+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:14.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:14.457+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:14.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:14.458+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:29:14.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:14.394+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:14.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:14.394+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:14.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:14.394+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:14.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:14.395+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:29:14.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:14.466+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:14.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:14.466+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:14.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:14.466+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:14.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:14.467+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:29:15.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:15.474+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:15.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:15.474+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:15.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:15.474+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:15.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:15.475+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:29:15.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:15.453+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:15.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:15.453+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:15.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:15.453+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:15.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:15.453+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:29:15.700 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:15.425+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:15.700 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:15.425+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:15.700 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:15.425+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:15.701 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:15.425+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:29:16.447 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:29:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:29:16.179+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 9291 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:29:16.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:16.395+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:16.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:16.395+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:16.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:16.395+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:16.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:16.396+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:29:16.449 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:16.409+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:16.449 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:16.409+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:16.449 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:16.409+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:16.450 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:16.411+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:29:16.799 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:16.479+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:16.799 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:16.479+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:16.799 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:16.480+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:16.800 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:16.481+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:29:17.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:17.369+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:17.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:17.369+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:17.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:17.369+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:17.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:17.369+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:29:17.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:17.437+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:17.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:17.437+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:17.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:17.437+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:17.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:17.438+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:29:17.699 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:17.500+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:17.699 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:17.500+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:17.699 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:17.500+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:17.699 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:17.501+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:29:18.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:18.338+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:18.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:18.338+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:18.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:18.338+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:18.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:18.339+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:29:18.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:18.525+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:18.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:18.525+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:18.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:18.525+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:18.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:18.525+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:29:18.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:18.442+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:18.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:18.442+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:18.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:18.442+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:18.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:18.443+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:29:19.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:19.437+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:19.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:19.437+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:19.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:19.437+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:19.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:19.437+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:29:19.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:19.305+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:19.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:19.305+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:19.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:19.305+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:19.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:19.305+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:29:19.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:19.476+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:19.699 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:19.476+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:19.699 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:19.476+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:19.699 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:19.477+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:29:20.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:20.258+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:20.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:20.258+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:20.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:20.258+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:20.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:20.259+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:29:20.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:20.482+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:20.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:20.482+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:20.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:20.482+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:20.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:20.483+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:29:20.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:20.416+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:20.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:20.416+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:20.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:20.416+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:20.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:20.417+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:29:21.447 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:29:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:29:21.180+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 9383 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:29:21.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:21.274+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:21.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:21.274+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:21.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:21.274+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:21.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:21.275+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:29:21.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:21.372+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:21.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:21.372+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:21.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:21.372+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:21.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:21.373+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:29:21.849 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:21.455+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:21.849 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:21.455+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:21.849 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:21.455+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:21.850 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:21.455+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:29:22.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:22.333+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:22.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:22.333+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:22.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:22.333+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:22.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:22.333+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:29:22.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:22.416+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:22.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:22.416+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:22.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:22.416+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:22.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:22.417+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:29:22.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:22.240+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:22.699 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:22.240+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:22.699 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:22.240+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:22.699 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:22.240+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:29:23.557 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:23.425+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:23.557 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:23.425+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:23.558 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:23.425+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:23.558 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:23.426+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:29:23.558 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:23.367+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:23.558 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:23.367+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:23.559 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:23.367+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:23.559 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:23.368+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:29:23.559 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:23.267+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:23.559 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:23.267+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:23.559 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:23.267+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:23.559 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:23.268+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:29:24.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:24.242+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:24.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:24.242+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:24.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:24.242+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:24.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:24.243+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:29:24.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:24.398+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:24.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:24.398+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:24.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:24.398+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:24.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:24.399+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:29:24.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:24.320+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:24.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:24.320+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:24.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:24.320+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:24.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:24.321+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:29:25.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:25.380+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:25.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:25.380+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:25.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:25.380+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:25.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:25.381+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:29:25.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:25.316+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:25.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:25.316+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:25.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:25.316+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:25.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:25.317+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:29:25.699 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:25.264+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:25.700 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:25.264+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:25.700 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:25.264+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:25.700 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:25.265+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:29:26.447 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:29:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:29:26.181+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 9474 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:29:26.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:26.284+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:26.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:26.284+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:26.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:26.284+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:26.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:26.285+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:29:26.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:26.313+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:26.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:26.313+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:26.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:26.313+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:26.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:26.314+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:29:26.449 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:26.404+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:26.449 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:26.404+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:26.449 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:26.404+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:26.449 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:26.406+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:29:27.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:27.295+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:27.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:27.295+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:27.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:27.295+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:27.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:27.296+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:29:27.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:27.305+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:27.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:27.305+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:27.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:27.305+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:27.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:27.306+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:29:27.699 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:27.427+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:27.699 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:27.427+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:27.699 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:27.427+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:27.699 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:27.428+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:29:28.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:28.306+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:28.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:28.306+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:28.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:28.306+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:28.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:28.307+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:29:28.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:28.468+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:28.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:28.468+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:28.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:28.468+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:28.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:28.469+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:29:28.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:28.355+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:28.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:28.355+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:28.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:28.355+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:28.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:28.356+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:29:29.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:29.337+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:29.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:29.337+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:29.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:29.337+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:29.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:29.338+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:29:29.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:29.277+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:29.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:29.277+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:29.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:29.277+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:29.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:29.278+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:29:29.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:29.432+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:29.699 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:29.432+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:29.699 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:29.432+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:29.699 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:29.432+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:29:30.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:30.294+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:30.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:30.294+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:30.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:30.294+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:30.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:30.295+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:29:30.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:30.400+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:30.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:30.400+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:30.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:30.400+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:30.699 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:30.401+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:29:30.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:30.381+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:30.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:30.381+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:30.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:30.381+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:30.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:30.382+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:29:31.447 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:29:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:29:31.182+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 9562 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:29:31.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:31.341+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:31.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:31.341+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:31.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:31.341+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:31.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:31.342+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:29:31.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:31.386+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:31.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:31.386+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:31.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:31.386+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:31.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:31.386+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:29:31.449 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:31.336+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:31.449 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:31.336+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:31.449 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:31.336+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:31.449 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:31.337+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:29:32.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:32.290+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:32.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:32.290+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:32.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:32.290+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:32.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:32.291+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:29:32.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:32.357+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:32.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:32.357+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:32.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:32.357+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:32.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:32.357+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:29:32.699 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:32.335+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:32.699 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:32.335+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:32.699 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:32.335+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:32.699 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:32.335+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:29:33.558 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:33.404+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:33.558 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:33.404+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:33.558 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:33.404+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:33.558 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:33.405+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:29:33.559 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:33.255+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:33.559 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:33.255+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:33.559 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:33.255+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:33.559 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:33.256+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:29:33.560 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:33.364+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:33.560 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:33.364+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:33.560 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:33.364+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:33.560 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:33.365+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:29:34.410 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:34.246+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:34.410 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:34.246+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:34.410 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:34.246+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:34.410 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:34.247+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:29:34.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:34.408+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:34.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:34.408+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:34.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:34.408+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:34.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:34.409+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:29:34.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:34.427+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:34.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:34.427+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:34.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:34.427+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:34.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:34.428+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:29:35.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:35.380+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:35.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:35.380+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:35.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:35.380+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:35.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:35.381+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:29:35.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:35.197+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:35.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:35.197+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:35.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:35.197+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:35.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:35.198+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:29:35.699 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:35.387+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:35.699 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:35.387+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:35.699 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:35.387+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:35.699 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:35.388+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:29:36.447 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:29:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:29:36.183+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 9650 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:29:36.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:36.156+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:36.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:36.156+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:36.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:36.156+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:36.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:36.157+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:29:36.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:36.375+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:36.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:36.375+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:36.449 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:36.375+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:36.449 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:36.376+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:29:36.449 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:36.387+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:36.449 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:36.387+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:36.449 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:36.387+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:36.449 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:36.387+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:29:37.370 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:37.173+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:37.371 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:37.173+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:37.371 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:37.173+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:37.371 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:37.174+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:29:37.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:37.369+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:37.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:37.369+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:37.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:37.369+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:37.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:37.370+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:29:37.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:37.430+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:37.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:37.430+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:37.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:37.430+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:37.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:37.430+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:29:38.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:38.371+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:38.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:38.371+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:38.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:38.371+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:38.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:38.372+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:29:38.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:38.160+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:38.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:38.160+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:38.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:38.160+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:38.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:38.161+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:29:38.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:38.474+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:38.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:38.474+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:38.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:38.474+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:38.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:38.475+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:29:39.428 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:39.152+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:39.428 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:39.152+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:39.428 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:39.152+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:39.429 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:39.153+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:29:39.429 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:39.327+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:39.429 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:39.327+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:39.429 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:39.327+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:39.429 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:39.328+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:29:39.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:39.427+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:39.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:39.427+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:39.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:39.427+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:39.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:39.428+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:29:40.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:40.363+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:40.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:40.363+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:40.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:40.363+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:40.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:40.363+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:29:40.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:40.388+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:40.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:40.388+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:40.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:40.388+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:40.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:40.388+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:29:40.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:40.196+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:40.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:40.196+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:40.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:40.196+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:40.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:40.196+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:29:41.447 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:29:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:29:41.184+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 9736 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:29:41.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:41.412+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:41.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:41.412+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:41.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:41.412+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:41.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:41.412+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:29:41.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:41.414+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:41.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:41.414+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:41.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:41.414+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:41.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:41.414+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:29:41.449 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:41.210+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:41.449 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:41.210+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:41.449 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:41.210+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:41.449 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:41.211+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:29:42.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:42.219+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:42.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:42.219+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:42.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:42.219+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:42.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:42.220+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:29:42.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:42.447+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:42.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:42.447+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:42.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:42.447+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:42.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:42.448+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:29:42.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:42.459+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:42.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:42.459+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:42.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:42.459+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:42.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:42.460+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:29:43.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:43.413+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:43.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:43.413+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:43.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:43.413+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:43.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:43.414+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:29:43.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:43.426+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:43.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:43.426+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:43.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:43.426+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:43.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:43.427+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:29:43.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:43.184+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:43.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:43.184+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:43.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:43.184+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:43.449 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:43.184+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:29:44.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:44.385+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:44.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:44.385+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:44.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:44.385+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:44.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:44.386+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:29:44.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:44.383+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:44.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:44.383+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:44.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:44.383+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:44.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:44.384+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:29:44.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:44.227+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:44.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:44.227+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:44.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:44.227+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:44.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:44.228+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:29:45.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:45.407+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:45.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:45.407+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:45.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:45.407+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:45.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:45.408+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:29:45.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:45.430+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:45.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:45.430+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:45.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:45.430+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:45.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:45.431+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:29:45.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:45.273+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:45.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:45.273+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:45.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:45.273+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:45.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:45.274+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:29:46.439 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:29:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:29:46.186+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 9830 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:29:46.440 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:46.416+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:46.440 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:46.416+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:46.440 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:46.416+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:46.440 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:46.418+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:29:46.440 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:46.302+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:46.441 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:46.302+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:46.441 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:46.302+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:46.441 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:46.303+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:29:46.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:46.438+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:46.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:46.438+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:46.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:46.438+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:46.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:46.439+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:29:47.375 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:47.279+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:47.376 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:47.279+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:47.376 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:47.279+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:47.376 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:47.279+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:29:47.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:47.374+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:47.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:47.374+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:47.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:47.374+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:47.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:47.375+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:29:47.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:47.456+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:47.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:47.456+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:47.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:47.456+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:47.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:47.458+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:29:48.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:48.502+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:48.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:48.502+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:48.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:48.502+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:48.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:48.502+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:29:48.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:48.401+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:48.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:48.401+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:48.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:48.401+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:48.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:48.402+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:29:48.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:48.290+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:48.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:48.290+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:48.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:48.290+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:48.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:48.290+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:29:49.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:49.506+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:49.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:49.506+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:49.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:49.506+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:49.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:49.506+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:29:49.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:49.378+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:49.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:49.378+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:49.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:49.378+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:49.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:49.379+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:29:49.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:49.310+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:49.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:49.310+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:49.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:49.310+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:49.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:49.310+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:29:50.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:50.420+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:50.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:50.420+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:50.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:50.420+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:50.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:50.422+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:29:50.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:50.505+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:50.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:50.505+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:50.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:50.505+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:50.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:50.506+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:29:50.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:50.329+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:50.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:50.329+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:50.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:50.329+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:50.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:50.329+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:29:51.447 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:29:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:29:51.188+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 9922 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:29:51.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:51.353+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:51.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:51.353+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:51.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:51.353+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:51.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:51.354+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:29:51.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:51.424+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:51.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:51.424+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:51.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:51.424+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:51.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:51.424+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:29:51.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:51.477+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:51.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:51.477+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:51.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:51.477+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:51.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:51.478+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:29:52.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:52.396+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:52.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:52.396+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:52.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:52.396+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:52.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:52.397+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:29:52.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:52.346+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:52.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:52.346+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:52.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:52.346+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:52.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:52.347+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:29:52.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:52.475+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:52.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:52.475+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:52.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:52.475+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:52.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:52.477+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:29:53.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:53.373+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:53.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:53.373+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:53.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:53.373+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:53.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:53.373+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:29:53.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:53.518+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:53.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:53.518+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:53.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:53.518+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:53.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:53.519+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:29:53.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:53.386+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:53.699 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:53.386+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:53.699 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:53.386+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:53.699 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:53.387+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:29:54.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:54.468+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:54.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:54.468+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:54.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:54.468+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:54.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:54.470+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:29:54.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:54.390+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:54.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:54.390+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:54.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:54.390+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:54.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:54.391+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:29:54.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:54.409+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:54.699 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:54.409+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:54.699 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:54.409+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:54.699 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:54.410+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:29:55.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:55.449+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:55.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:55.449+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:55.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:55.449+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:55.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:55.450+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:29:55.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:55.425+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:55.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:55.425+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:55.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:55.425+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:55.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:55.425+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:29:55.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:55.430+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:55.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:55.430+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:55.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:55.430+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:55.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:55.430+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:29:56.447 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:29:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:29:56.188+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 10019 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:29:56.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:56.425+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:56.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:56.425+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:56.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:56.425+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:56.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:56.425+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:29:56.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:56.492+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:56.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:56.492+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:56.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:56.492+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:56.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:56.492+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:29:56.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:56.463+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:56.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:56.463+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:56.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:56.463+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:56.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:56.464+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:29:57.506 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:57.448+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:57.506 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:57.448+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:57.506 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:57.448+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:57.506 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:57.448+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:29:57.507 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:57.404+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:57.507 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:57.404+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:57.507 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:57.404+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:57.507 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:57.404+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:29:57.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:57.504+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:57.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:57.504+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:57.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:57.504+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:57.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:57.505+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:29:58.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:58.416+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:58.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:58.416+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:58.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:58.416+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:58.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:58.417+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:29:58.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:58.529+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:58.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:58.529+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:58.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:58.529+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:58.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:58.530+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:29:58.699 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:58.469+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:58.699 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:58.469+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:58.699 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:58.469+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:58.699 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:58.470+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:29:59.833 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:59.451+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:59.833 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:59.451+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:59.833 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:59.451+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:29:59.833 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:29:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:29:59.452+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:29:59.833 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:59.450+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:59.834 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:59.450+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:59.834 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:59.450+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:29:59.834 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:29:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:29:59.451+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:29:59.834 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:59.574+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:59.834 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:59.574+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:59.834 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:59.574+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:29:59.835 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:29:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:29:59.575+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:30:00.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:00.472+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:00.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:00.472+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:00.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:00.472+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:00.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:00.472+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:30:00.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:00.405+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:00.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:00.405+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:00.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:00.405+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:00.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:00.406+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:30:00.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:00.566+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:00.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:00.566+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:00.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:00.566+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:00.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:00.566+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:30:01.447 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:30:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:30:01.189+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 10109 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:30:01.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:01.552+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:01.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:01.552+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:01.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:01.552+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:01.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:01.553+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:30:01.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:01.451+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:01.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:01.451+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:01.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:01.451+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:01.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:01.452+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:30:01.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:01.446+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:01.949 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:01.446+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:01.949 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:01.446+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:01.949 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:01.447+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:30:02.577 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:02.440+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:02.578 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:02.440+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:02.578 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:02.440+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:02.578 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:02.441+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:30:02.578 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:02.398+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:02.578 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:02.398+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:02.578 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:02.398+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:02.579 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:02.399+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:30:02.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:02.576+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:02.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:02.576+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:02.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:02.576+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:02.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:02.577+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:30:03.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:03.533+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:03.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:03.533+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:03.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:03.533+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:03.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:03.533+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:30:03.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:03.411+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:03.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:03.411+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:03.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:03.411+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:03.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:03.411+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:30:03.699 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:03.356+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:03.699 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:03.356+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:03.699 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:03.356+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:03.699 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:03.357+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:30:04.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:04.373+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:04.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:04.373+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:04.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:04.373+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:04.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:04.374+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:30:04.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:04.405+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:04.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:04.405+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:04.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:04.405+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:04.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:04.406+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:30:04.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:04.494+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:04.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:04.494+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:04.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:04.494+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:04.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:04.494+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:30:05.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:05.535+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:05.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:05.535+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:05.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:05.535+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:05.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:05.535+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:30:05.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:05.365+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:05.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:05.365+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:05.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:05.365+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:05.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:05.366+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:30:05.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:05.405+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:05.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:05.405+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:05.699 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:05.405+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:05.699 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:05.406+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:30:06.447 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:30:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:30:06.190+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 10206 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:30:06.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:06.416+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:06.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:06.416+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:06.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:06.416+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:06.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:06.417+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:30:06.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:06.396+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:06.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:06.396+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:06.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:06.396+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:06.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:06.397+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:30:06.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:06.510+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:06.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:06.510+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:06.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:06.510+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:06.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:06.511+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:30:07.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:07.448+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:07.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:07.448+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:07.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:07.448+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:07.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:07.449+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:30:07.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:07.552+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:07.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:07.552+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:07.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:07.552+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:07.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:07.553+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:30:07.699 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:07.362+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:07.699 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:07.362+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:07.699 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:07.362+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:07.699 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:07.363+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:30:08.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:08.545+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:08.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:08.545+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:08.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:08.545+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:08.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:08.545+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:30:08.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:08.360+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:08.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:08.360+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:08.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:08.360+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:08.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:08.361+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:30:08.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:08.479+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:08.699 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:08.479+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:08.699 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:08.479+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:08.699 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:08.480+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:30:09.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:09.470+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:09.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:09.470+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:09.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:09.470+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:09.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:09.471+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:30:09.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:09.371+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:09.699 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:09.371+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:09.699 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:09.371+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:09.699 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:09.372+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:30:09.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:09.531+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:09.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:09.531+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:09.700 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:09.531+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:09.700 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:09.532+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:30:10.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:10.456+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:10.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:10.456+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:10.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:10.456+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:10.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:10.457+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:30:10.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:10.521+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:10.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:10.521+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:10.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:10.521+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:10.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:10.522+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:30:10.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:10.324+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:10.699 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:10.324+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:10.699 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:10.324+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:10.699 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:10.325+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:30:11.447 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:30:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:30:11.192+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 10300 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:30:11.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:11.287+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:11.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:11.287+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:11.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:11.287+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:11.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:11.288+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:30:11.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:11.465+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:11.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:11.465+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:11.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:11.465+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:11.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:11.467+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:30:11.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:11.548+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:11.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:11.548+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:11.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:11.548+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:11.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:11.549+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:30:12.576 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:12.320+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:12.577 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:12.320+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:12.577 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:12.320+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:12.577 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:12.321+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:30:12.577 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:12.492+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:12.577 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:12.492+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:12.578 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:12.492+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:12.578 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:12.493+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:30:12.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:12.575+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:12.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:12.575+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:12.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:12.575+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:12.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:12.576+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:30:13.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:13.575+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:13.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:13.575+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:13.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:13.575+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:13.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:13.576+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:30:13.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:13.353+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:13.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:13.353+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:13.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:13.353+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:13.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:13.354+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:30:13.699 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:13.444+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:13.699 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:13.444+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:13.699 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:13.444+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:13.699 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:13.445+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:30:14.610 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:14.444+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:14.611 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:14.444+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:14.611 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:14.444+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:14.611 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:14.445+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:30:14.611 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:14.331+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:14.611 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:14.331+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:14.611 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:14.331+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:14.611 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:14.332+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:30:14.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:14.609+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:14.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:14.609+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:14.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:14.609+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:14.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:14.610+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:30:15.648 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:15.413+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:15.649 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:15.413+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:15.649 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:15.413+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:15.649 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:15.414+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:30:15.649 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:15.313+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:15.649 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:15.313+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:15.650 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:15.313+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:15.650 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:15.313+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:30:15.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:15.647+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:15.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:15.647+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:15.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:15.647+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:15.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:15.647+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:30:16.447 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:30:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:30:16.193+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 10397 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:30:16.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:16.363+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:16.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:16.363+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:16.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:16.363+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:16.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:16.364+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:30:16.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:16.407+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:16.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:16.407+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:16.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:16.407+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:16.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:16.408+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:30:16.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:16.612+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:16.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:16.612+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:16.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:16.612+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:16.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:16.613+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:30:17.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:17.391+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:17.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:17.391+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:17.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:17.391+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:17.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:17.392+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:30:17.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:17.400+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:17.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:17.400+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:17.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:17.400+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:17.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:17.401+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:30:17.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:17.585+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:17.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:17.585+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:17.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:17.585+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:17.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:17.587+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:30:18.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:18.430+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:18.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:18.430+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:18.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:18.430+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:18.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:18.431+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:30:18.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:18.594+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:18.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:18.594+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:18.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:18.594+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:18.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:18.595+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:30:18.699 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:18.424+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:18.699 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:18.424+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:18.699 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:18.424+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:18.699 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:18.425+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:30:19.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:19.409+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:19.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:19.409+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:19.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:19.409+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:19.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:19.411+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:30:19.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:19.418+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:19.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:19.418+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:19.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:19.418+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:19.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:19.419+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:30:19.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:19.634+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:19.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:19.634+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:19.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:19.634+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:19.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:19.635+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:30:20.650 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:20.391+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:20.650 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:20.391+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:20.651 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:20.391+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:20.651 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:20.392+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:30:20.651 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:20.374+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:20.651 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:20.374+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:20.651 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:20.374+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:20.651 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:20.375+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:30:20.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:20.649+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:20.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:20.649+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:20.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:20.649+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:20.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:20.650+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:30:21.447 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:30:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:30:21.194+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 10491 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:30:21.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:21.393+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:21.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:21.393+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:21.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:21.393+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:21.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:21.393+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:30:21.449 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:21.370+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:21.449 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:21.370+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:21.449 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:21.370+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:21.449 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:21.371+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:30:21.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:21.649+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:21.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:21.649+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:21.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:21.649+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:21.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:21.649+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:30:22.691 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:22.430+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:22.691 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:22.430+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:22.691 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:22.430+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:22.692 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:22.431+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:30:22.692 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:22.360+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:22.692 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:22.360+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:22.692 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:22.360+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:22.692 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:22.362+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:30:22.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:22.690+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:22.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:22.690+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:22.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:22.690+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:22.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:22.691+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:30:23.687 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:23.364+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:23.687 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:23.364+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:23.687 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:23.364+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:23.687 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:23.365+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:30:23.688 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:23.394+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:23.688 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:23.394+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:23.688 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:23.394+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:23.688 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:23.395+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:30:23.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:23.685+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:23.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:23.685+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:23.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:23.685+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:23.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:23.687+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:30:24.695 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:24.434+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:24.695 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:24.434+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:24.695 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:24.434+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:24.696 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:24.435+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:30:24.696 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:24.372+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:24.696 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:24.372+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:24.696 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:24.372+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:24.696 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:24.373+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:30:24.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:24.694+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:24.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:24.694+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:24.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:24.694+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:24.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:24.695+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:30:25.687 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:25.368+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:25.687 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:25.368+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:25.688 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:25.368+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:25.688 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:25.369+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:30:25.688 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:25.461+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:25.688 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:25.461+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:25.688 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:25.461+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:25.688 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:25.462+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:30:25.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:25.686+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:25.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:25.686+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:25.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:25.686+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:25.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:25.687+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:30:26.447 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:30:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:30:26.195+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 10588 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:30:26.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:26.339+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:26.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:26.339+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:26.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:26.339+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:26.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:26.339+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:30:26.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:26.464+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:26.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:26.464+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:26.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:26.464+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:26.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:26.464+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:30:26.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:26.647+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:26.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:26.647+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:26.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:26.647+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:26.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:26.648+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:30:27.682 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:27.510+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:27.682 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:27.510+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:27.682 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:27.510+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:27.682 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:27.510+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:30:27.682 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:27.297+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:27.682 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:27.297+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:27.683 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:27.297+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:27.683 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:27.299+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:30:27.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:27.680+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:27.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:27.680+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:27.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:27.680+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:27.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:27.681+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:30:28.671 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:28.482+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:28.671 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:28.482+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:28.671 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:28.482+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:28.671 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:28.482+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:30:28.672 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:28.298+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:28.672 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:28.298+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:28.672 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:28.298+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:28.672 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:28.299+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:30:28.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:28.670+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:28.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:28.670+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:28.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:28.670+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:28.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:28.671+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:30:29.644 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:29.283+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:29.645 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:29.283+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:29.645 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:29.283+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:29.645 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:29.284+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:30:29.645 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:29.478+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:29.645 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:29.478+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:29.645 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:29.478+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:29.645 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:29.478+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:30:29.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:29.643+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:29.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:29.643+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:29.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:29.643+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:29.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:29.644+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:30:30.597 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:30.274+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:30.597 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:30.274+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:30.597 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:30.274+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:30.597 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:30.276+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:30:30.598 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:30.517+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:30.598 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:30.517+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:30.598 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:30.517+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:30.598 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:30.517+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:30:30.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:30.595+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:30.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:30.595+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:30.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:30.595+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:30.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:30.596+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:30:31.471 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:30:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:30:31.197+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 10682 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:30:31.471 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:31.235+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:31.471 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:31.235+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:31.471 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:31.235+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:31.472 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:31.236+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:30:31.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:31.469+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:31.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:31.469+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:31.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:31.469+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:31.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:31.470+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:30:31.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:31.601+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:31.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:31.601+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:31.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:31.601+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:31.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:31.602+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:30:32.588 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:32.270+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:32.588 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:32.270+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:32.588 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:32.270+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:32.588 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:32.271+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:30:32.589 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:32.498+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:32.589 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:32.498+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:32.589 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:32.498+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:32.589 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:32.498+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:30:32.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:32.640+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:32.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:32.640+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:32.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:32.640+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:32.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:32.641+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:30:33.679 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:33.318+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:33.679 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:33.318+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:33.680 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:33.318+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:33.680 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:33.318+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:30:33.680 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:33.508+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:33.680 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:33.508+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:33.680 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:33.508+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:33.680 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:33.509+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:30:33.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:33.678+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:33.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:33.678+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:33.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:33.678+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:33.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:33.678+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:30:34.676 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:34.348+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:34.676 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:34.348+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:34.676 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:34.348+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:34.676 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:34.348+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:30:34.677 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:34.468+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:34.677 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:34.468+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:34.677 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:34.468+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:34.677 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:34.469+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:30:34.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:34.674+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:34.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:34.674+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:34.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:34.674+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:34.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:34.675+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:30:35.688 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:35.384+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:35.689 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:35.384+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:35.689 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:35.384+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:35.689 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:35.384+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:30:35.689 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:35.478+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:35.689 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:35.478+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:35.689 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:35.478+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:35.690 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:35.478+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:30:35.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:35.687+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:35.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:35.687+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:35.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:35.687+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:35.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:35.688+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:30:36.523 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:30:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:30:36.199+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 10779 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:30:36.523 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:36.420+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:36.524 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:36.420+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:36.524 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:36.420+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:36.524 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:36.421+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:30:36.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:36.522+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:36.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:36.522+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:36.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:36.522+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:36.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:36.523+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:30:36.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:36.695+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:36.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:36.695+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:36.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:36.695+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:36.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:36.696+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:30:37.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:37.504+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:37.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:37.504+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:37.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:37.504+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:37.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:37.504+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:30:37.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:37.398+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:37.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:37.398+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:37.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:37.398+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:37.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:37.399+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:30:38.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:37.746+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:38.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:37.746+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:38.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:37.746+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:38.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:37.746+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:30:38.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:38.466+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:38.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:38.466+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:38.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:38.466+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:38.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:38.467+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:30:38.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:38.392+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:38.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:38.392+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:38.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:38.392+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:38.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:38.392+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:30:39.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:38.729+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:39.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:38.729+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:39.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:38.729+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:39.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:38.730+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:30:39.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:39.461+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:39.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:39.461+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:39.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:39.461+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:39.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:39.462+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:30:39.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:39.368+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:39.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:39.368+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:39.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:39.368+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:39.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:39.369+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:30:40.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:39.729+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:40.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:39.729+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:40.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:39.729+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:40.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:39.730+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:30:40.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:40.373+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:40.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:40.373+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:40.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:40.373+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:40.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:40.373+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:30:40.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:40.425+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:40.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:40.425+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:40.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:40.425+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:40.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:40.425+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:30:41.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:40.700+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:41.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:40.700+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:41.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:40.700+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:41.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:40.701+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:30:41.454 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:30:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:30:41.199+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 10874 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:30:41.454 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:41.385+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:41.454 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:41.385+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:41.455 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:41.385+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:41.455 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:41.386+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:30:41.715 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:41.452+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:41.715 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:41.452+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:41.715 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:41.452+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:41.715 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:41.453+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:30:42.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:41.713+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:42.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:41.713+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:42.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:41.713+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:42.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:41.714+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:30:42.694 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:42.442+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:42.694 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:42.442+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:42.694 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:42.442+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:42.694 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:42.443+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:30:42.695 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:42.433+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:42.695 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:42.433+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:42.695 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:42.433+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:42.695 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:42.434+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:30:42.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:42.702+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:42.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:42.702+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:42.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:42.702+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:42.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:42.702+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:30:43.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:43.425+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:43.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:43.425+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:43.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:43.425+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:43.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:43.426+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:30:43.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:43.453+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:43.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:43.453+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:43.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:43.453+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:43.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:43.454+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:30:44.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:43.703+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:44.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:43.703+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:44.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:43.703+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:44.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:43.704+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:30:44.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:44.430+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:44.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:44.430+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:44.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:44.430+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:44.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:44.431+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:30:44.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:44.501+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:44.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:44.501+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:44.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:44.501+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:44.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:44.502+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:30:45.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:44.725+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:45.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:44.725+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:45.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:44.725+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:45.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:44.726+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:30:45.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:45.510+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:45.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:45.510+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:45.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:45.510+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:45.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:45.511+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:30:45.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:45.440+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:45.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:45.440+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:45.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:45.440+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:45.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:45.441+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:30:46.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:45.746+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:46.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:45.746+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:46.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:45.746+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:46.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:45.747+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:30:46.469 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:30:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:30:46.200+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 10972 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:30:46.775 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:46.468+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:46.775 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:46.468+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:46.775 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:46.468+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:46.775 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:46.469+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:30:46.775 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:46.468+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:46.775 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:46.468+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:46.776 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:46.468+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:46.776 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:46.469+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:30:47.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:46.773+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:47.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:46.773+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:47.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:46.773+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:47.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:46.774+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:30:47.744 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:47.462+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:47.744 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:47.462+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:47.744 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:47.462+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:47.745 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:47.463+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:30:47.745 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:47.479+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:47.745 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:47.479+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:47.745 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:47.479+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:47.746 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:47.480+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:30:48.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:47.742+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:48.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:47.742+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:48.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:47.742+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:48.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:47.744+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:30:48.729 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:48.450+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:48.729 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:48.450+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:48.729 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:48.450+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:48.729 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:48.451+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:30:48.729 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:48.485+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:48.730 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:48.485+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:48.730 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:48.485+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:48.730 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:48.486+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:30:49.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:48.727+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:49.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:48.727+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:49.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:48.727+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:49.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:48.728+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:30:49.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:49.535+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:49.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:49.535+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:49.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:49.535+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:49.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:49.536+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:30:49.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:49.426+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:49.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:49.426+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:49.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:49.426+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:49.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:49.427+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:30:50.142 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:49.757+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:50.142 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:49.757+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:50.142 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:49.757+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:50.142 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:49.758+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:30:50.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:50.492+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:50.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:50.492+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:50.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:50.492+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:50.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:50.493+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:30:50.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:50.410+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:50.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:50.410+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:50.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:50.410+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:50.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:50.411+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:30:51.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:50.737+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:51.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:50.737+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:51.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:50.737+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:51.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:50.738+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:30:51.527 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:30:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:30:51.202+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 11067 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:30:51.528 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:51.365+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:51.528 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:51.365+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:51.528 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:51.365+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:51.528 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:51.365+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:30:51.785 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:51.526+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:51.786 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:51.526+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:51.786 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:51.526+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:51.786 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:51.527+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:30:52.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:51.784+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:52.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:51.784+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:52.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:51.784+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:52.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:51.785+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:30:52.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:52.358+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:52.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:52.358+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:52.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:52.358+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:52.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:52.359+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:30:52.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:52.489+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:52.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:52.489+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:52.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:52.489+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:52.699 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:52.490+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:30:53.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:52.744+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:53.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:52.744+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:53.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:52.744+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:53.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:52.745+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:30:53.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:53.402+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:53.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:53.402+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:53.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:53.402+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:53.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:53.403+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:30:53.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:53.524+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:53.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:53.524+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:53.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:53.525+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:53.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:53.525+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:30:54.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:53.736+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:54.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:53.736+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:54.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:53.736+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:54.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:53.737+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:30:54.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:54.534+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:54.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:54.534+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:54.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:54.534+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:54.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:54.535+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:30:54.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:54.413+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:54.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:54.413+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:54.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:54.413+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:54.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:54.413+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:30:55.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:54.706+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:55.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:54.706+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:55.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:54.706+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:55.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:54.707+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:30:55.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:55.574+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:55.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:55.574+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:55.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:55.574+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:55.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:55.575+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:30:55.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:55.396+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:55.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:55.396+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:55.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:55.396+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:55.699 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:55.397+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:30:56.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:55.742+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:56.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:55.742+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:56.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:55.742+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:56.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:55.743+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:30:56.606 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:30:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:30:56.203+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 11165 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:30:56.606 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:56.396+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:56.607 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:56.396+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:56.607 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:56.396+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:56.607 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:56.397+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:30:56.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:56.604+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:56.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:56.604+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:56.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:56.604+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:56.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:56.606+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:30:56.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:56.759+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:56.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:56.759+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:56.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:56.759+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:56.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:56.760+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:30:57.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:57.420+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:57.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:57.420+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:57.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:57.420+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:57.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:57.421+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:30:57.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:57.560+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:57.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:57.560+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:57.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:57.560+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:57.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:57.561+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:30:58.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:57.712+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:58.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:57.712+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:58.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:57.712+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:58.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:57.713+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:30:58.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:58.382+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:58.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:58.382+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:58.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:58.382+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:58.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:58.383+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:30:58.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:58.527+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:58.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:58.527+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:58.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:58.527+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:58.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:58.527+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:30:59.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:58.735+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:59.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:58.735+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:59.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:58.735+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:30:59.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:58.736+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:30:59.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:59.366+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:59.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:59.366+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:59.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:59.366+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:30:59.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:30:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:30:59.367+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:30:59.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:59.543+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:59.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:59.543+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:59.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:59.543+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:30:59.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:30:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:30:59.543+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:31:00.142 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:59.780+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:00.142 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:59.780+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:00.143 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:59.780+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:00.143 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:30:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:30:59.781+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:31:00.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:00.567+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:00.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:00.567+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:00.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:00.567+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:00.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:00.568+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:31:00.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:00.400+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:00.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:00.400+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:00.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:00.400+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:00.699 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:00.401+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:31:01.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:00.778+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:01.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:00.778+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:01.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:00.778+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:01.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:00.778+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:31:01.568 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:31:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:31:01.204+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 11260 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:31:01.569 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:01.447+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:01.569 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:01.447+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:01.569 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:01.447+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:01.569 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:01.448+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:31:01.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:01.567+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:01.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:01.567+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:01.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:01.567+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:01.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:01.568+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:31:01.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:01.748+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:01.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:01.748+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:01.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:01.748+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:01.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:01.749+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:31:02.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:02.568+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:02.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:02.568+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:02.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:02.568+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:02.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:02.570+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:31:02.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:02.428+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:02.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:02.428+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:02.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:02.428+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:02.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:02.429+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:31:03.163 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:02.780+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:03.164 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:02.780+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:03.164 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:02.780+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:03.164 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:02.781+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:31:03.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:03.409+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:03.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:03.409+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:03.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:03.409+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:03.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:03.409+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:31:03.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:03.547+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:03.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:03.547+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:03.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:03.547+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:03.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:03.548+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:31:03.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:03.775+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:03.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:03.775+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:03.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:03.775+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:03.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:03.776+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:31:04.788 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:04.538+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:04.788 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:04.538+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:04.788 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:04.538+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:04.788 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:04.539+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:31:04.789 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:04.456+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:04.789 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:04.456+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:04.789 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:04.456+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:04.789 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:04.457+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:31:05.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:04.786+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:05.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:04.786+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:05.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:04.786+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:05.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:04.787+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:31:05.780 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:05.468+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:05.780 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:05.468+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:05.780 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:05.468+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:05.780 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:05.468+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:31:05.781 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:05.568+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:05.781 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:05.568+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:05.781 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:05.568+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:05.781 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:05.569+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:31:06.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:05.778+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:06.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:05.778+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:06.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:05.778+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:06.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:05.779+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:31:06.541 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:06.448+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:06.542 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:06.449+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:06.542 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:06.449+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:06.542 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:06.450+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:31:06.542 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:31:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:31:06.206+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 11358 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:31:06.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:06.540+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:06.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:06.540+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:06.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:06.540+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:06.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:06.541+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:31:06.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:06.775+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:06.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:06.775+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:06.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:06.775+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:06.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:06.775+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:31:07.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:07.415+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:07.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:07.415+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:07.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:07.415+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:07.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:07.416+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:31:07.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:07.516+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:07.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:07.516+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:07.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:07.516+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:07.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:07.517+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:31:08.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:07.727+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:08.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:07.727+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:08.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:07.727+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:08.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:07.728+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:31:08.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:08.399+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:08.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:08.399+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:08.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:08.399+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:08.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:08.399+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:31:08.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:08.564+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:08.699 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:08.564+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:08.699 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:08.564+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:08.699 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:08.564+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:31:09.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:08.778+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:09.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:08.778+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:09.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:08.778+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:09.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:08.779+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:31:09.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:09.560+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:09.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:09.560+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:09.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:09.560+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:09.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:09.560+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:31:09.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:09.443+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:09.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:09.443+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:09.699 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:09.443+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:09.699 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:09.444+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:31:10.143 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:09.747+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:10.143 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:09.747+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:10.143 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:09.747+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:10.144 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:09.748+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:31:10.795 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:10.447+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:10.796 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:10.447+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:10.796 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:10.447+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:10.796 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:10.447+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:31:10.797 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:10.521+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:10.797 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:10.521+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:10.797 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:10.521+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:10.797 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:10.522+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:31:11.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:10.793+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:11.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:10.793+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:11.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:10.793+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:11.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:10.795+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:31:11.482 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:31:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:31:11.207+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 11453 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:31:11.482 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:11.449+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:11.482 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:11.449+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:11.482 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:11.449+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:11.483 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:11.450+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:31:11.767 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:11.481+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:11.767 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:11.481+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:11.768 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:11.481+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:11.768 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:11.482+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:31:12.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:11.766+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:12.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:11.766+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:12.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:11.766+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:12.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:11.767+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:31:12.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:12.470+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:12.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:12.470+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:12.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:12.470+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:12.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:12.471+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:31:12.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:12.411+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:12.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:12.411+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:12.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:12.411+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:12.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:12.412+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:31:13.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:12.770+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:13.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:12.770+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:13.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:12.770+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:13.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:12.771+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:31:13.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:13.429+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:13.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:13.429+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:13.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:13.429+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:13.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:13.430+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:31:13.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:13.362+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:13.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:13.362+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:13.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:13.362+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:13.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:13.363+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:31:14.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:13.733+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:14.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:13.733+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:14.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:13.733+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:14.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:13.734+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:31:14.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:14.411+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:14.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:14.411+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:14.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:14.411+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:14.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:14.412+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:31:14.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:14.366+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:14.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:14.366+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:14.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:14.366+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:14.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:14.366+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:31:15.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:14.701+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:15.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:14.701+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:15.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:14.701+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:15.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:14.701+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:31:15.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:15.430+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:15.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:15.430+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:15.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:15.430+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:15.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:15.431+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:31:15.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:15.379+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:15.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:15.379+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:15.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:15.379+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:15.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:15.380+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:31:16.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:15.707+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:16.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:15.707+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:16.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:15.707+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:16.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:15.708+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:31:16.697 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:31:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:31:16.208+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 11551 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:31:16.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:16.430+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:16.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:16.430+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:16.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:16.430+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:16.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:16.431+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:31:16.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:16.417+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:16.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:16.417+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:16.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:16.417+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:16.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:16.417+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:31:17.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:16.705+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:17.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:16.705+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:17.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:16.705+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:17.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:16.706+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:31:17.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:17.391+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:17.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:17.391+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:17.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:17.391+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:17.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:17.392+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:31:17.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:17.425+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:17.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:17.425+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:17.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:17.425+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:17.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:17.426+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:31:18.047 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:17.718+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:18.048 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:17.718+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:18.048 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:17.718+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:18.048 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:17.719+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:31:18.677 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:18.374+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:18.677 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:18.374+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:18.677 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:18.374+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:18.677 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:18.375+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:31:18.678 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:18.443+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:18.678 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:18.443+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:18.678 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:18.443+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:18.678 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:18.443+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:31:18.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:18.675+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:18.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:18.675+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:18.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:18.675+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:18.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:18.676+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:31:19.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:19.453+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:19.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:19.453+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:19.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:19.453+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:19.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:19.454+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:31:19.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:19.370+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:19.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:19.370+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:19.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:19.370+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:19.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:19.371+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:31:20.144 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:19.725+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:20.144 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:19.725+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:20.144 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:19.725+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:20.144 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:19.726+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:31:20.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:20.413+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:20.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:20.413+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:20.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:20.413+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:20.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:20.413+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:31:20.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:20.496+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:20.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:20.496+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:20.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:20.496+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:20.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:20.497+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:31:21.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:20.771+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:21.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:20.771+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:21.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:20.771+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:21.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:20.771+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:31:21.503 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:31:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:31:21.210+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 11646 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:31:21.503 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:21.410+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:21.504 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:21.410+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:21.504 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:21.410+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:21.504 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:21.410+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:31:21.805 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:21.503+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:21.805 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:21.503+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:21.805 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:21.503+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:21.805 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:21.503+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:31:22.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:21.805+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:22.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:21.805+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:22.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:21.805+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:22.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:21.805+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:31:22.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:22.524+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:22.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:22.524+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:22.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:22.524+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:22.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:22.525+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:31:22.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:22.435+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:22.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:22.435+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:22.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:22.435+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:22.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:22.436+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:31:23.100 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:22.802+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:23.100 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:22.802+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:23.100 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:22.802+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:23.101 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:22.803+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:31:23.817 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:23.478+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:23.817 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:23.478+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:23.817 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:23.478+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:23.817 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:23.479+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:31:23.818 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:23.462+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:23.818 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:23.462+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:23.818 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:23.462+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:23.818 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:23.463+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:31:24.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:23.816+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:24.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:23.816+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:24.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:23.816+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:24.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:23.818+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:31:24.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:24.469+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:24.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:24.469+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:24.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:24.469+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:24.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:24.469+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:31:24.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:24.433+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:24.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:24.433+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:24.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:24.433+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:24.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:24.434+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:31:25.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:24.867+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:25.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:24.867+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:25.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:24.867+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:25.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:24.867+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:31:25.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:25.429+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:25.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:25.429+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:25.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:25.429+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:25.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:25.430+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:31:25.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:25.466+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:25.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:25.466+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:25.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:25.466+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:25.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:25.466+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:31:26.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:25.877+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:26.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:25.877+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:26.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:25.877+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:26.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:25.878+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:31:26.463 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:31:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:31:26.212+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 11744 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:31:26.463 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:26.461+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:26.464 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:26.461+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:26.464 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:26.461+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:26.464 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:26.462+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:31:26.891 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:26.470+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:26.891 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:26.470+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:26.891 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:26.470+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:26.891 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:26.471+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:31:27.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:26.890+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:27.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:26.890+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:27.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:26.890+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:27.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:26.890+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:31:27.862 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:27.482+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:27.862 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:27.482+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:27.862 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:27.482+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:27.863 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:27.482+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:31:27.865 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:27.483+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:27.865 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:27.483+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:27.866 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:27.483+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:27.866 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:27.484+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:31:28.153 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:27.861+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:28.153 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:27.861+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:28.154 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:27.861+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:28.154 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:27.862+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:31:28.857 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:28.506+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:28.857 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:28.506+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:28.857 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:28.506+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:28.858 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:28.507+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:31:28.858 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:28.451+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:28.858 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:28.451+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:28.858 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:28.451+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:28.858 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:28.452+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:31:29.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:28.856+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:29.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:28.856+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:29.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:28.856+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:29.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:28.857+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:31:29.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:29.424+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:29.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:29.424+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:29.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:29.424+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:29.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:29.426+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:31:29.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:29.538+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:29.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:29.538+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:29.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:29.538+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:29.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:29.539+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:31:30.144 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:29.809+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:30.144 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:29.809+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:30.144 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:29.809+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:30.145 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:29.810+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:31:30.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:30.380+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:30.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:30.380+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:30.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:30.380+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:30.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:30.381+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:31:30.793 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:30.536+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:30.793 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:30.536+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:30.794 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:30.536+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:30.794 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:30.536+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:31:31.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:30.792+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:31.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:30.792+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:31.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:30.792+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:31.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:30.793+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:31:31.526 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:31.381+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:31.526 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:31.381+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:31.527 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:31.381+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:31.527 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:31.382+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:31:31.527 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:31:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:31:31.213+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 11839 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:31:31.790 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:31.525+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:31.790 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:31.525+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:31.790 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:31.525+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:31.790 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:31.526+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:31:32.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:31.788+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:32.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:31.788+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:32.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:31.788+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:32.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:31.789+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:31:32.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:32.476+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:32.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:32.476+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:32.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:32.476+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:32.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:32.477+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:31:32.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:32.348+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:32.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:32.348+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:32.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:32.348+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:32.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:32.349+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:31:33.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:32.770+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:33.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:32.770+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:33.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:32.770+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:33.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:32.771+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:31:33.564 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:33.441+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:33.564 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:33.441+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:33.564 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:33.441+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:33.564 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:33.442+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:31:33.564 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:33.318+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:33.565 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:33.318+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:33.565 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:33.318+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:33.565 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:33.319+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:31:33.883 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:33.729+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:33.883 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:33.729+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:33.883 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:33.729+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:33.883 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:33.730+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:31:34.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:34.306+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:34.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:34.306+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:34.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:34.306+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:34.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:34.307+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:31:34.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:34.480+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:34.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:34.480+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:34.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:34.480+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:34.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:34.481+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:31:35.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:34.779+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:35.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:34.779+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:35.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:34.779+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:35.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:34.780+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:31:35.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:35.310+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:35.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:35.310+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:35.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:35.310+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:35.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:35.311+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:31:35.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:35.525+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:35.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:35.525+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:35.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:35.525+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:35.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:35.525+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:31:36.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:35.744+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:36.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:35.744+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:36.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:35.744+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:36.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:35.745+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:31:36.533 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:36.276+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:36.534 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:36.276+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:36.534 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:36.276+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:36.534 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:36.277+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:31:36.534 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:31:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:31:36.214+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 11937 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:31:36.785 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:36.532+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:36.785 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:36.532+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:36.785 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:36.532+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:36.786 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:36.533+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:31:37.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:36.784+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:37.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:36.784+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:37.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:36.784+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:37.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:36.785+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:31:37.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:37.536+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:37.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:37.536+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:37.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:37.536+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:37.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:37.537+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:31:37.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:37.307+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:37.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:37.307+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:37.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:37.307+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:37.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:37.307+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:31:38.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:37.774+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:38.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:37.774+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:38.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:37.774+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:38.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:37.775+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:31:38.563 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:38.303+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:38.564 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:38.303+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:38.564 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:38.303+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:38.564 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:38.303+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:31:38.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:38.562+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:38.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:38.562+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:38.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:38.562+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:38.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:38.563+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:31:38.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:38.802+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:38.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:38.802+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:38.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:38.802+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:38.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:38.803+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:31:39.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:39.585+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:39.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:39.585+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:39.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:39.585+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:39.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:39.586+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:31:39.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:39.348+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:39.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:39.348+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:39.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:39.348+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:39.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:39.349+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:31:40.144 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:39.835+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:40.145 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:39.835+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:40.145 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:39.835+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:40.145 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:39.836+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:31:40.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:40.338+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:40.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:40.338+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:40.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:40.338+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:40.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:40.339+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:31:40.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:40.630+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:40.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:40.630+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:40.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:40.630+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:40.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:40.631+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:31:40.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:40.851+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:40.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:40.851+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:40.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:40.851+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:40.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:40.852+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:31:41.590 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:31:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:31:41.216+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 12031 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:31:41.590 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:41.362+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:41.590 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:41.362+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:41.591 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:41.362+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:41.591 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:41.363+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:31:41.850 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:41.588+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:41.850 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:41.588+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:41.850 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:41.588+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:41.851 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:41.589+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:31:42.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:41.849+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:42.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:41.849+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:42.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:41.849+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:42.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:41.850+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:31:42.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:42.556+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:42.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:42.556+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:42.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:42.556+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:42.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:42.556+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:31:42.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:42.413+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:42.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:42.413+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:42.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:42.413+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:42.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:42.414+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:31:43.099 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:42.844+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:43.099 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:42.844+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:43.099 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:42.844+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:43.100 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:42.845+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:31:43.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:43.399+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:43.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:43.399+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:43.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:43.399+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:43.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:43.400+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:31:43.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:43.567+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:43.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:43.567+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:43.699 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:43.567+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:43.699 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:43.568+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:31:44.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:43.872+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:44.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:43.872+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:44.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:43.872+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:44.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:43.872+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:31:44.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:44.383+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:44.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:44.383+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:44.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:44.383+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:44.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:44.384+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:31:44.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:44.539+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:44.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:44.539+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:44.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:44.539+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:44.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:44.540+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:31:45.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:44.876+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:45.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:44.876+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:45.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:44.876+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:45.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:44.878+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:31:45.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:45.501+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:45.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:45.501+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:45.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:45.501+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:45.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:45.502+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:31:45.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:45.372+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:45.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:45.372+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:45.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:45.372+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:45.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:45.373+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:31:46.172 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:45.879+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:46.172 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:45.879+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:46.172 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:45.879+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:46.173 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:45.880+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:31:46.447 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:31:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:31:46.217+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 12129 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:31:46.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:46.382+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:46.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:46.382+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:46.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:46.382+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:46.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:46.382+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:31:46.852 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:46.533+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:46.852 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:46.533+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:46.852 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:46.533+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:46.852 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:46.534+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:31:47.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:46.850+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:47.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:46.850+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:47.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:46.850+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:47.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:46.851+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:31:47.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:47.583+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:47.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:47.583+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:47.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:47.583+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:47.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:47.584+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:31:47.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:47.404+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:47.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:47.404+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:47.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:47.404+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:47.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:47.405+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:31:48.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:47.816+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:48.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:47.816+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:48.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:47.816+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:48.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:47.817+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:31:48.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:48.538+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:48.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:48.538+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:48.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:48.538+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:48.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:48.539+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:31:48.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:48.450+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:48.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:48.450+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:48.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:48.450+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:48.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:48.451+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:31:49.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:48.830+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:49.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:48.830+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:49.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:48.830+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:49.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:48.832+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:31:49.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:49.527+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:49.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:49.527+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:49.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:49.527+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:49.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:49.528+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:31:49.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:49.425+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:49.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:49.425+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:49.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:49.425+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:49.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:49.426+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:31:50.145 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:49.783+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:50.145 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:49.783+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:50.146 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:49.783+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:50.146 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:49.784+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:31:50.802 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:50.546+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:50.802 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:50.546+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:50.803 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:50.546+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:50.803 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:50.546+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:31:50.803 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:50.456+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:50.803 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:50.456+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:50.803 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:50.456+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:50.803 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:50.457+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:31:51.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:50.800+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:51.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:50.800+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:51.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:50.800+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:51.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:50.802+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:31:51.500 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:31:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:31:51.219+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 12224 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:31:51.500 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:51.467+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:51.501 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:51.467+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:51.501 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:51.467+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:51.501 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:51.468+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:31:51.793 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:51.499+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:51.793 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:51.499+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:51.793 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:51.499+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:51.793 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:51.500+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:31:52.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:51.791+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:52.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:51.791+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:52.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:51.791+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:52.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:51.792+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:31:52.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:52.528+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:52.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:52.528+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:52.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:52.528+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:52.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:52.529+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:31:52.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:52.501+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:52.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:52.501+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:52.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:52.501+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:52.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:52.502+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:31:53.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:52.754+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:53.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:52.754+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:53.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:52.754+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:53.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:52.755+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:31:53.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:53.540+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:53.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:53.540+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:53.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:53.540+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:53.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:53.541+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:31:53.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:53.562+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:53.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:53.562+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:53.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:53.562+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:53.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:53.563+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:31:54.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:53.719+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:54.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:53.719+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:54.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:53.719+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:54.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:53.719+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:31:54.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:54.522+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:54.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:54.522+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:54.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:54.522+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:54.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:54.523+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:31:54.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:54.765+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:54.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:54.765+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:54.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:54.765+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:54.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:54.766+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:31:54.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:54.555+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:54.949 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:54.555+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:54.949 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:54.555+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:54.949 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:54.556+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:31:55.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:55.556+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:55.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:55.556+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:55.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:55.556+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:55.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:55.557+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:31:55.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:55.556+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:55.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:55.556+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:55.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:55.556+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:55.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:55.557+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:31:56.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:55.788+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:56.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:55.788+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:56.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:55.788+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:56.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:55.789+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:31:56.528 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:31:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:31:56.220+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 12322 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:31:56.799 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:56.533+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:56.799 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:56.533+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:56.799 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:56.533+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:56.799 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:56.534+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:31:56.800 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:56.527+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:56.800 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:56.527+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:56.800 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:56.527+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:56.800 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:56.528+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:31:57.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:56.798+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:57.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:56.798+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:57.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:56.798+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:57.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:56.798+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:31:57.834 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:57.565+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:57.834 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:57.565+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:57.834 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:57.565+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:57.834 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:57.566+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:31:57.835 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:57.556+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:57.835 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:57.556+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:57.835 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:57.556+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:57.835 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:57.557+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:31:58.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:57.832+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:58.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:57.832+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:58.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:57.832+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:58.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:57.833+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:31:58.810 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:58.532+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:58.810 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:58.532+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:58.811 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:58.532+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:58.811 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:58.533+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:31:58.811 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:58.602+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:58.811 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:58.602+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:58.811 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:58.602+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:58.811 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:58.602+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:31:59.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:58.809+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:59.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:58.809+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:59.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:58.809+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:31:59.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:58.809+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:31:59.791 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:59.634+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:59.792 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:59.634+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:59.792 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:59.634+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:31:59.792 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:31:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:31:59.635+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:31:59.792 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:59.485+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:59.792 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:59.485+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:59.793 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:59.485+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:31:59.793 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:31:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:31:59.485+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:32:00.145 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:59.790+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:00.146 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:59.790+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:00.146 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:59.790+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:00.146 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:31:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:31:59.791+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:32:00.780 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:00.641+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:00.780 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:00.641+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:00.780 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:00.641+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:00.780 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:00.642+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:32:00.780 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:00.522+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:00.781 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:00.522+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:00.781 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:00.522+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:00.781 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:00.523+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:32:01.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:00.778+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:01.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:00.778+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:01.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:00.778+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:01.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:00.779+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:32:01.531 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:32:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:32:01.222+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 12416 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:32:01.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:01.665+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:01.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:01.665+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:01.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:01.665+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:01.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:01.666+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:32:01.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:01.572+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:01.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:01.572+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:01.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:01.572+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:01.949 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:01.573+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:32:01.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:01.744+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:01.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:01.744+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:01.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:01.744+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:01.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:01.745+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:32:02.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:02.695+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:02.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:02.695+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:02.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:02.695+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:02.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:02.696+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:32:02.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:02.621+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:02.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:02.621+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:02.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:02.621+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:02.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:02.622+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:32:02.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:02.730+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:02.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:02.730+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:02.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:02.730+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:02.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:02.731+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:32:03.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:03.745+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:03.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:03.745+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:03.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:03.745+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:03.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:03.746+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:32:03.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:03.701+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:03.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:03.701+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:03.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:03.701+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:03.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:03.702+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:32:03.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:03.635+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:03.949 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:03.635+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:03.949 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:03.635+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:03.949 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:03.636+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:32:04.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:04.684+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:04.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:04.684+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:04.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:04.684+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:04.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:04.685+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:32:04.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:04.720+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:04.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:04.720+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:04.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:04.720+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:04.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:04.721+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:32:04.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:04.656+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:04.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:04.656+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:04.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:04.656+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:04.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:04.657+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:32:05.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:05.707+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:05.955 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:05.707+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:05.955 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:05.707+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:05.955 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:05.708+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:32:05.956 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:05.702+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:05.956 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:05.702+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:05.956 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:05.702+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:05.956 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:05.703+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:32:05.957 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:05.613+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:05.957 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:05.613+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:05.957 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:05.613+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:05.957 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:05.614+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:32:06.632 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:32:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:32:06.223+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 12512 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:32:06.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:06.710+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:06.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:06.710+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:06.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:06.710+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:06.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:06.711+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:32:06.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:06.699+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:06.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:06.699+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:06.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:06.699+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:06.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:06.700+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:32:06.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:06.631+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:06.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:06.631+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:06.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:06.631+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:06.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:06.631+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:32:07.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:07.708+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:07.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:07.708+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:07.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:07.708+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:07.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:07.709+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:32:07.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:07.678+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:07.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:07.678+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:07.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:07.678+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:07.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:07.679+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:32:07.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:07.652+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:07.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:07.652+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:07.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:07.652+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:07.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:07.653+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:32:08.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:08.727+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:08.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:08.727+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:08.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:08.727+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:08.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:08.728+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:32:08.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:08.715+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:08.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:08.715+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:08.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:08.715+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:08.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:08.716+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:32:08.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:08.635+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:08.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:08.635+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:08.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:08.635+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:08.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:08.636+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:32:09.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:09.618+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:09.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:09.618+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:09.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:09.618+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:09.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:09.619+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:32:09.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:09.762+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:09.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:09.762+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:09.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:09.762+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:09.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:09.763+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:32:09.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:09.739+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:09.949 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:09.739+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:09.949 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:09.739+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:09.949 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:09.740+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:32:10.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:10.654+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:10.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:10.654+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:10.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:10.654+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:10.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:10.655+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:32:10.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:10.735+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:10.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:10.735+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:10.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:10.735+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:10.949 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:10.735+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:32:10.949 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:10.753+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:10.949 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:10.753+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:10.949 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:10.753+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:10.949 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:10.754+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:32:11.670 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:32:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:32:11.224+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 12606 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:32:11.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:11.713+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:11.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:11.713+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:11.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:11.713+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:11.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:11.714+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:32:11.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:11.784+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:11.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:11.784+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:11.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:11.784+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:11.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:11.785+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:32:11.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:11.668+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:11.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:11.668+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:11.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:11.668+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:11.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:11.669+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:32:12.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:12.695+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:12.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:12.695+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:12.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:12.695+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:12.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:12.695+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:32:12.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:12.681+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:12.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:12.681+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:12.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:12.681+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:12.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:12.682+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:32:12.949 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:12.768+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:12.949 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:12.768+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:12.949 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:12.768+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:12.949 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:12.770+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:32:13.819 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:13.649+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:13.820 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:13.649+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:13.820 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:13.649+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:13.820 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:13.651+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:32:13.820 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:13.707+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:13.820 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:13.707+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:13.820 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:13.707+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:13.821 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:13.708+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:32:13.821 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:13.750+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:13.821 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:13.750+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:13.821 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:13.750+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:13.821 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:13.751+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:32:14.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:14.704+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:14.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:14.704+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:14.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:14.704+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:14.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:14.705+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:32:14.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:14.763+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:14.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:14.763+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:14.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:14.763+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:14.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:14.764+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:32:14.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:14.695+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:14.949 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:14.695+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:14.949 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:14.695+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:14.949 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:14.697+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:32:16.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:15.738+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:16.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:15.738+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:16.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:15.738+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:16.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:15.739+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:32:16.199 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:15.797+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:16.199 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:15.797+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:16.199 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:15.797+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:16.199 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:15.798+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:32:16.199 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:15.722+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:16.199 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:15.722+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:16.200 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:15.722+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:16.200 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:15.723+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:32:16.697 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:32:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:32:16.226+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 12703 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:32:17.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:16.735+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:17.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:16.735+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:17.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:16.735+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:17.199 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:16.736+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:32:17.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:16.782+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:17.200 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:16.782+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:17.200 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:16.782+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:17.200 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:16.783+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:32:17.200 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:16.837+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:17.201 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:16.837+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:17.201 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:16.837+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:17.201 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:16.838+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:32:18.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:17.704+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:18.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:17.704+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:18.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:17.704+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:18.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:17.705+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:32:18.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:17.779+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:18.201 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:17.779+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:18.201 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:17.779+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:18.202 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:17.779+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:32:18.202 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:17.831+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:18.202 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:17.831+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:18.202 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:17.831+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:18.202 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:17.831+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:32:18.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:18.822+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:18.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:18.822+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:18.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:18.822+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:18.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:18.823+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:32:18.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:18.666+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:18.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:18.666+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:18.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:18.666+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:18.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:18.667+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:32:18.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:18.806+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:18.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:18.806+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:18.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:18.806+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:18.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:18.807+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:32:19.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:19.690+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:19.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:19.690+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:19.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:19.690+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:19.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:19.691+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:32:19.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:19.857+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:19.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:19.857+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:19.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:19.857+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:19.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:19.857+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:32:19.949 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:19.794+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:19.949 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:19.794+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:19.949 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:19.794+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:19.949 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:19.795+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:32:20.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:20.814+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:20.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:20.814+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:20.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:20.814+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:20.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:20.815+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:32:20.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:20.770+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:20.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:20.770+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:20.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:20.770+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:20.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:20.771+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:32:20.949 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:20.650+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:20.949 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:20.650+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:20.949 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:20.650+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:20.949 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:20.651+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:32:21.616 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:32:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:32:21.227+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 12797 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:32:21.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:21.819+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:21.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:21.819+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:21.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:21.819+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:21.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:21.819+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:32:21.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:21.787+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:21.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:21.787+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:21.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:21.787+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:21.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:21.787+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:32:21.949 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:21.614+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:21.949 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:21.614+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:21.949 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:21.614+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:21.949 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:21.616+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:32:22.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:22.803+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:22.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:22.803+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:22.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:22.803+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:22.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:22.804+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:32:22.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:22.784+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:22.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:22.784+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:22.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:22.784+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:22.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:22.785+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:32:22.949 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:22.592+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:22.949 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:22.592+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:22.949 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:22.592+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:22.949 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:22.593+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:32:23.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:23.764+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:23.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:23.764+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:23.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:23.764+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:23.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:23.765+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:32:23.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:23.620+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:23.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:23.620+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:23.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:23.620+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:23.949 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:23.621+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:32:23.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:23.797+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:23.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:23.797+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:23.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:23.797+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:23.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:23.797+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:32:24.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:24.732+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:24.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:24.732+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:24.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:24.732+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:24.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:24.733+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:32:24.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:24.666+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:24.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:24.666+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:24.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:24.666+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:24.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:24.667+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:32:24.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:24.765+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:24.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:24.765+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:24.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:24.765+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:24.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:24.766+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:32:25.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:25.780+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:25.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:25.780+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:25.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:25.780+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:25.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:25.781+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:32:25.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:25.663+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:25.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:25.663+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:25.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:25.663+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:25.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:25.663+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:32:25.949 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:25.741+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:25.949 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:25.741+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:25.949 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:25.741+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:25.949 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:25.742+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:32:26.447 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:32:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:32:26.229+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 12894 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:32:26.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:26.745+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:26.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:26.745+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:26.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:26.745+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:26.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:26.745+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:32:26.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:26.685+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:26.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:26.685+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:26.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:26.685+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:26.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:26.685+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:32:26.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:26.768+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:26.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:26.768+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:26.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:26.768+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:26.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:26.769+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:32:27.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:27.743+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:27.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:27.743+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:27.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:27.743+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:27.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:27.744+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:32:27.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:27.708+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:27.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:27.708+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:27.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:27.708+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:27.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:27.709+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:32:27.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:27.653+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:27.949 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:27.653+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:27.949 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:27.653+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:27.949 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:27.653+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:32:28.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:28.754+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:28.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:28.754+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:28.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:28.754+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:28.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:28.755+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:32:28.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:28.712+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:28.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:28.712+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:28.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:28.712+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:28.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:28.713+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:32:28.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:28.656+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:28.949 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:28.656+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:28.949 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:28.656+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:28.949 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:28.657+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:32:29.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:29.733+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:29.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:29.733+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:29.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:29.733+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:29.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:29.733+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:32:29.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:29.668+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:29.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:29.668+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:29.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:29.668+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:29.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:29.669+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:32:29.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:29.697+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:29.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:29.697+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:29.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:29.697+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:29.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:29.698+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:32:30.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:30.636+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:30.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:30.636+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:30.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:30.636+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:30.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:30.637+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:32:30.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:30.654+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:30.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:30.654+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:30.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:30.654+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:30.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:30.654+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:32:30.949 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:30.775+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:30.949 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:30.775+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:30.949 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:30.775+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:30.949 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:30.776+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:32:31.623 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:32:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:32:31.230+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 12988 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:32:31.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:31.735+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:31.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:31.735+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:31.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:31.735+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:31.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:31.735+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:32:31.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:31.622+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:31.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:31.622+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:31.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:31.622+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:31.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:31.623+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:32:31.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:31.665+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:31.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:31.665+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:31.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:31.665+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:31.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:31.667+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:32:32.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:32.597+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:32.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:32.597+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:32.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:32.597+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:32.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:32.598+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:32:32.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:32.702+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:32.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:32.702+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:32.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:32.702+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:32.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:32.703+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:32:32.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:32.674+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:32.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:32.674+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:32.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:32.674+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:32.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:32.675+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:32:33.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:33.621+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:33.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:33.621+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:33.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:33.621+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:33.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:33.622+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:32:33.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:33.689+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:33.949 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:33.689+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:33.949 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:33.689+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:33.949 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:33.690+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:32:33.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:33.653+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:33.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:33.653+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:33.950 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:33.653+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:33.950 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:33.653+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:32:34.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:34.625+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:34.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:34.625+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:34.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:34.625+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:34.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:34.626+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:32:34.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:34.630+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:34.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:34.630+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:34.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:34.630+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:34.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:34.631+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:32:34.949 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:34.702+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:34.949 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:34.702+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:34.949 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:34.702+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:34.949 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:34.703+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:32:35.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:35.651+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:35.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:35.651+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:35.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:35.651+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:35.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:35.653+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:32:35.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:35.753+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:35.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:35.753+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:35.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:35.753+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:35.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:35.753+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:32:35.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:35.602+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:35.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:35.602+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:35.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:35.602+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:35.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:35.602+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:32:36.559 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:32:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:32:36.232+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 13085 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:32:36.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:36.619+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:36.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:36.619+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:36.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:36.619+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:36.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:36.619+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:32:36.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:36.761+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:36.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:36.761+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:36.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:36.761+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:36.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:36.762+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:32:36.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:36.557+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:36.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:36.557+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:36.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:36.557+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:36.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:36.558+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:32:37.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:37.806+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:37.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:37.806+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:37.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:37.806+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:37.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:37.807+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:32:37.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:37.594+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:37.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:37.594+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:37.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:37.594+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:37.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:37.595+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:32:37.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:37.565+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:37.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:37.565+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:37.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:37.565+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:37.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:37.567+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:32:38.790 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:38.522+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:38.790 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:38.522+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:38.790 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:38.522+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:38.790 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:38.523+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:32:38.791 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:38.566+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:38.791 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:38.566+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:38.791 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:38.566+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:38.791 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:38.567+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:32:39.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:38.789+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:39.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:38.789+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:39.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:38.789+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:39.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:38.789+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:32:39.832 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:39.575+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:39.833 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:39.575+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:39.833 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:39.575+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:39.833 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:39.576+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:32:39.833 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:39.793+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:39.833 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:39.793+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:39.834 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:39.793+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:39.834 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:39.794+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:32:39.834 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:39.566+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:39.834 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:39.566+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:39.834 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:39.566+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:39.834 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:39.567+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:32:40.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:40.755+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:40.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:40.755+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:40.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:40.755+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:40.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:40.756+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:32:40.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:40.534+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:40.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:40.534+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:40.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:40.534+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:40.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:40.535+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:32:40.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:40.527+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:40.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:40.527+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:40.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:40.527+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:40.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:40.529+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:32:41.522 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:32:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:32:41.233+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 13179 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:32:41.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:41.749+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:41.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:41.749+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:41.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:41.749+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:41.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:41.749+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:32:41.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:41.537+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:41.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:41.537+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:41.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:41.537+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:41.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:41.537+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:32:41.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:41.520+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:41.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:41.520+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:41.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:41.520+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:41.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:41.521+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:32:42.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:42.505+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:42.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:42.505+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:42.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:42.505+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:42.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:42.505+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:32:42.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:42.742+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:42.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:42.742+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:42.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:42.742+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:42.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:42.743+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:32:42.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:42.526+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:42.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:42.526+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:42.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:42.526+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:42.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:42.527+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:32:43.764 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:43.476+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:43.764 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:43.476+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:43.765 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:43.476+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:43.765 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:43.477+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:32:43.765 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:43.563+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:43.765 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:43.563+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:43.765 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:43.563+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:43.765 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:43.563+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:32:44.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:43.763+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:44.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:43.763+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:44.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:43.763+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:44.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:43.763+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:32:44.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:44.736+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:44.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:44.736+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:44.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:44.736+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:44.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:44.737+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:32:44.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:44.523+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:44.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:44.523+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:44.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:44.524+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:44.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:44.524+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:32:44.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:44.605+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:44.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:44.605+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:44.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:44.605+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:44.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:44.606+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:32:45.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:45.743+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:45.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:45.743+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:45.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:45.743+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:45.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:45.744+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:32:45.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:45.531+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:45.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:45.531+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:45.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:45.531+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:45.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:45.531+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:32:45.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:45.574+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:45.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:45.574+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:45.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:45.574+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:45.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:45.575+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:32:46.487 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:32:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:32:46.235+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 13276 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:32:46.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:46.486+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:46.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:46.486+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:46.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:46.486+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:46.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:46.487+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:32:46.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:46.715+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:46.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:46.715+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:46.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:46.715+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:46.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:46.716+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:32:46.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:46.541+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:46.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:46.541+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:46.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:46.541+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:46.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:46.542+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:32:47.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:47.727+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:47.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:47.727+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:47.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:47.727+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:47.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:47.728+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:32:47.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:47.504+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:47.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:47.504+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:47.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:47.504+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:47.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:47.505+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:32:47.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:47.515+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:47.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:47.515+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:47.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:47.515+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:47.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:47.515+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:32:48.748 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:48.456+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:48.748 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:48.456+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:48.748 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:48.456+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:48.748 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:48.457+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:32:48.749 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:48.474+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:48.749 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:48.474+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:48.749 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:48.474+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:48.749 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:48.475+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:32:49.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:48.746+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:49.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:48.746+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:49.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:48.746+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:49.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:48.747+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:32:49.832 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:49.489+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:49.833 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:49.489+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:49.833 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:49.489+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:49.833 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:49.490+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:32:49.833 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:49.486+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:49.833 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:49.486+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:49.834 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:49.486+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:49.834 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:49.487+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:32:49.834 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:49.718+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:49.834 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:49.718+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:49.834 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:49.718+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:49.834 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:49.719+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:32:50.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:50.445+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:50.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:50.445+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:50.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:50.445+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:50.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:50.446+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:32:50.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:50.462+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:50.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:50.462+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:50.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:50.462+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:50.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:50.462+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:32:51.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:50.721+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:51.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:50.721+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:51.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:50.721+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:51.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:50.722+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:32:51.494 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:32:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:32:51.236+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 13370 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:32:51.495 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:51.462+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:51.495 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:51.462+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:51.495 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:51.462+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:51.495 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:51.463+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:32:51.747 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:51.493+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:51.747 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:51.493+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:51.747 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:51.493+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:51.748 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:51.494+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:32:52.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:51.746+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:52.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:51.746+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:52.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:51.746+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:52.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:51.746+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:32:52.797 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:52.507+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:52.797 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:52.507+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:52.797 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:52.507+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:52.797 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:52.510+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:32:52.797 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:52.455+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:52.798 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:52.455+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:52.798 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:52.455+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:52.798 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:52.455+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:32:53.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:52.796+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:53.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:52.796+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:53.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:52.796+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:53.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:52.797+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:32:53.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:53.413+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:53.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:53.413+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:53.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:53.413+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:53.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:53.413+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:32:53.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:53.535+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:53.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:53.535+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:53.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:53.535+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:53.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:53.536+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:32:54.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:53.830+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:54.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:53.830+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:54.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:53.830+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:54.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:53.831+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:32:54.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:54.415+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:54.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:54.415+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:54.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:54.415+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:54.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:54.415+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:32:54.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:54.530+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:54.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:54.530+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:54.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:54.530+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:54.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:54.531+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:32:55.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:54.807+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:55.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:54.807+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:55.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:54.807+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:55.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:54.807+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:32:55.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:55.368+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:55.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:55.368+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:55.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:55.368+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:55.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:55.369+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:32:55.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:55.547+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:55.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:55.547+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:55.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:55.547+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:55.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:55.548+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:32:56.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:55.800+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:56.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:55.800+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:56.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:55.800+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:56.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:55.801+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:32:56.577 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:32:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:32:56.239+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 13467 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:32:56.577 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:56.324+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:56.577 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:56.324+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:56.578 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:56.324+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:56.578 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:56.325+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:32:56.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:56.576+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:56.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:56.576+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:56.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:56.576+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:56.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:56.577+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:32:56.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:56.754+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:56.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:56.754+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:56.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:56.754+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:56.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:56.754+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:32:57.546 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:57.287+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:57.547 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:57.287+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:57.547 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:57.287+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:57.547 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:57.288+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:32:57.803 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:57.545+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:57.803 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:57.545+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:57.803 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:57.545+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:57.803 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:57.546+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:32:58.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:57.801+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:58.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:57.801+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:58.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:57.801+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:58.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:57.802+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:32:58.534 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:58.267+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:58.535 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:58.267+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:58.535 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:58.267+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:58.535 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:58.268+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:32:58.808 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:58.533+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:58.808 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:58.533+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:58.808 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:58.533+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:58.809 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:58.534+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:32:59.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:58.806+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:59.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:58.806+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:59.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:58.806+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:32:59.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:58.808+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:32:59.503 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:59.232+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:59.503 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:59.232+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:59.503 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:59.232+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:32:59.503 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:32:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:32:59.233+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:32:59.794 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:59.501+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:59.794 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:59.501+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:59.794 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:59.501+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:32:59.794 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:32:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:32:59.503+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:33:00.149 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:59.792+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:00.149 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:59.792+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:00.149 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:59.792+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:00.150 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:32:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:32:59.794+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:33:00.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:00.257+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:00.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:00.257+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:00.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:00.257+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:00.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:00.257+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:33:00.786 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:00.468+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:00.786 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:00.468+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:00.786 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:00.468+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:00.787 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:00.468+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:33:01.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:00.784+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:01.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:00.784+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:01.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:00.784+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:01.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:00.785+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:33:01.486 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:33:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:33:01.239+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 13561 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:33:01.487 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:01.231+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:01.487 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:01.231+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:01.487 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:01.231+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:01.487 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:01.232+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:33:01.751 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:01.485+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:01.752 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:01.485+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:01.752 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:01.485+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:01.752 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:01.486+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:33:02.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:01.750+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:02.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:01.750+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:02.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:01.750+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:02.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:01.751+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:33:02.521 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:02.235+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:02.522 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:02.235+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:02.522 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:02.235+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:02.522 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:02.236+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:33:02.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:02.713+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:02.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:02.713+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:02.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:02.713+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:02.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:02.714+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:33:02.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:02.520+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:02.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:02.520+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:02.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:02.520+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:02.949 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:02.521+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:33:03.549 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:03.246+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:03.549 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:03.246+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:03.549 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:03.246+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:03.550 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:03.247+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:33:03.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:03.548+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:03.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:03.548+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:03.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:03.548+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:03.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:03.548+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:33:03.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:03.743+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:03.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:03.743+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:03.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:03.743+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:03.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:03.744+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:33:04.555 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:04.294+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:04.555 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:04.294+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:04.555 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:04.294+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:04.556 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:04.295+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:33:04.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:04.713+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:04.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:04.713+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:04.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:04.713+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:04.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:04.714+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:33:04.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:04.554+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:04.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:04.554+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:04.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:04.554+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:04.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:04.555+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:33:05.560 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:05.307+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:05.560 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:05.307+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:05.561 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:05.307+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:05.561 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:05.307+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:33:05.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:05.674+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:05.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:05.674+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:05.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:05.674+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:05.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:05.675+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:33:05.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:05.559+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:05.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:05.559+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:05.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:05.559+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:05.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:05.560+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:33:06.306 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:33:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:33:06.241+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 13658 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:33:06.610 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:06.305+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:06.611 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:06.305+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:06.611 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:06.305+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:06.611 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:06.306+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:33:06.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:06.609+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:06.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:06.609+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:06.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:06.609+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:06.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:06.610+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:33:06.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:06.673+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:06.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:06.673+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:06.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:06.673+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:06.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:06.674+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:33:07.562 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:07.268+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:07.563 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:07.268+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:07.563 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:07.268+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:07.563 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:07.269+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:33:07.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:07.561+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:07.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:07.561+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:07.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:07.561+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:07.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:07.562+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:33:07.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:07.652+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:07.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:07.652+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:07.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:07.652+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:07.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:07.652+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:33:08.608 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:08.537+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:08.608 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:08.537+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:08.608 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:08.537+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:08.608 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:08.538+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:33:08.608 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:08.305+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:08.609 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:08.305+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:08.609 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:08.305+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:08.609 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:08.306+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:33:08.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:08.606+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:08.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:08.606+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:08.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:08.606+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:08.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:08.607+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:33:09.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:09.274+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:09.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:09.274+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:09.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:09.274+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:09.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:09.274+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:33:09.833 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:09.610+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:09.833 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:09.610+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:09.833 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:09.610+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:09.833 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:09.611+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:33:09.833 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:09.524+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:09.834 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:09.524+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:09.834 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:09.524+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:09.834 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:09.524+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:33:10.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:10.311+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:10.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:10.311+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:10.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:10.311+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:10.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:10.313+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:33:10.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:10.640+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:10.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:10.640+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:10.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:10.640+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:10.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:10.641+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:33:10.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:10.507+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:10.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:10.507+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:10.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:10.507+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:10.949 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:10.508+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:33:11.506 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:33:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:33:11.242+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 13752 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:33:11.507 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:11.343+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:11.507 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:11.343+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:11.507 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:11.343+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:11.507 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:11.344+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:33:11.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:11.505+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:11.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:11.505+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:11.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:11.505+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:11.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:11.506+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:33:11.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:11.598+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:11.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:11.598+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:11.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:11.599+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:11.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:11.599+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:33:12.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:12.377+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:12.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:12.377+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:12.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:12.377+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:12.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:12.377+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:33:12.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:12.626+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:12.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:12.626+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:12.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:12.626+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:12.949 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:12.628+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:33:12.949 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:12.501+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:12.950 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:12.501+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:12.950 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:12.501+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:12.950 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:12.501+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:33:13.662 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:13.512+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:13.662 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:13.512+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:13.662 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:13.512+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:13.662 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:13.513+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:33:13.663 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:13.387+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:13.663 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:13.387+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:13.663 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:13.387+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:13.663 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:13.388+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:33:13.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:13.660+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:13.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:13.660+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:13.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:13.660+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:13.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:13.662+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:33:14.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:14.358+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:14.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:14.358+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:14.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:14.358+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:14.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:14.358+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:33:14.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:14.498+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:14.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:14.498+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:14.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:14.498+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:14.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:14.499+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:33:14.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:14.689+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:14.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:14.689+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:14.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:14.689+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:14.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:14.690+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:33:15.513 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:15.313+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:15.513 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:15.313+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:15.513 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:15.313+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:15.513 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:15.314+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:33:15.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:15.661+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:15.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:15.661+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:15.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:15.661+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:15.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:15.662+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:33:15.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:15.512+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:15.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:15.512+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:15.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:15.512+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:15.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:15.512+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:33:16.539 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:33:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:33:16.244+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 13849 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:33:16.540 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:16.285+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:16.540 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:16.285+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:16.540 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:16.285+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:16.540 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:16.285+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:33:16.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:16.701+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:16.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:16.701+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:16.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:16.701+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:16.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:16.702+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:33:16.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:16.538+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:16.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:16.538+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:16.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:16.538+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:16.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:16.539+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:33:17.582 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:17.268+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:17.583 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:17.268+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:17.583 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:17.268+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:17.583 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:17.269+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:33:17.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:17.580+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:17.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:17.580+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:17.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:17.580+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:17.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:17.581+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:33:17.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:17.750+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:17.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:17.750+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:17.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:17.750+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:17.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:17.751+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:33:18.614 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:18.262+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:18.614 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:18.262+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:18.615 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:18.262+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:18.615 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:18.263+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:33:18.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:18.613+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:18.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:18.613+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:18.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:18.613+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:18.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:18.614+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:33:18.949 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:18.774+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:18.949 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:18.774+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:18.950 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:18.774+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:18.950 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:18.775+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:33:19.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:19.220+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:19.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:19.220+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:19.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:19.220+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:19.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:19.220+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:33:19.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:19.790+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:19.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:19.790+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:19.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:19.790+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:19.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:19.791+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:33:19.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:19.641+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:19.949 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:19.641+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:19.949 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:19.641+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:19.949 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:19.642+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:33:20.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:20.178+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:20.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:20.178+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:20.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:20.178+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:20.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:20.178+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:33:20.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:20.667+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:20.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:20.667+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:20.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:20.667+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:20.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:20.668+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:33:20.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:20.832+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:20.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:20.832+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:20.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:20.832+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:20.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:20.832+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:33:21.447 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:33:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:33:21.245+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 13943 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:33:21.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:21.154+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:21.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:21.154+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:21.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:21.154+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:21.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:21.154+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:33:21.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:21.682+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:21.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:21.682+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:21.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:21.682+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:21.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:21.683+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:33:21.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:21.835+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:21.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:21.835+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:21.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:21.835+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:21.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:21.835+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:33:22.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:22.141+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:22.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:22.141+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:22.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:22.141+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:22.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:22.141+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:33:23.165 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:22.787+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:23.166 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:22.787+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:23.166 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:22.787+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:23.166 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:22.788+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:33:23.166 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:22.721+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:23.166 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:22.721+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:23.167 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:22.721+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:23.167 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:22.722+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:33:23.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:23.164+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:23.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:23.164+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:23.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:23.164+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:23.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:23.165+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:33:23.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:23.738+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:23.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:23.738+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:23.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:23.738+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:23.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:23.739+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:33:23.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:23.698+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:23.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:23.698+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:23.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:23.698+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:23.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:23.698+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:33:24.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:24.127+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:24.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:24.127+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:24.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:24.127+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:24.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:24.128+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:33:24.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:24.723+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:24.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:24.723+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:24.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:24.723+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:24.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:24.723+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:33:24.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:24.659+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:24.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:24.659+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:24.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:24.659+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:24.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:24.660+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:33:25.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:25.116+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:25.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:25.116+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:25.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:25.116+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:25.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:25.117+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:33:26.107 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:25.705+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:26.107 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:25.705+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:26.107 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:25.705+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:26.107 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:25.706+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:33:26.107 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:25.711+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:26.108 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:25.711+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:26.108 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:25.711+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:26.108 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:25.711+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:33:26.371 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:33:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:33:26.247+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 14040 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:33:26.372 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:26.105+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:26.372 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:26.105+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:26.372 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:26.105+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:26.372 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:26.106+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:33:27.069 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:26.697+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:27.069 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:26.697+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:27.070 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:26.697+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:27.070 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:26.698+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:33:27.070 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:26.729+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:27.070 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:26.729+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:27.070 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:26.729+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:27.070 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:26.729+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:33:27.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:27.068+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:27.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:27.068+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:27.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:27.068+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:27.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:27.069+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:33:27.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:27.677+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:27.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:27.677+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:27.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:27.677+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:27.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:27.678+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:33:27.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:27.766+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:27.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:27.766+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:27.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:27.766+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:27.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:27.767+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:33:28.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:28.072+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:28.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:28.072+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:28.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:28.072+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:28.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:28.073+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:33:29.119 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:28.736+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:29.119 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:28.736+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:29.119 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:28.736+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:29.119 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:28.737+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:33:29.119 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:28.706+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:29.120 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:28.706+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:29.120 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:28.706+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:29.120 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:28.706+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:33:29.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:29.117+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:29.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:29.117+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:29.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:29.117+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:29.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:29.118+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:33:29.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:29.703+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:29.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:29.703+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:29.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:29.703+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:29.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:29.704+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:33:29.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:29.690+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:29.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:29.690+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:29.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:29.690+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:29.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:29.690+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:33:30.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:30.082+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:30.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:30.082+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:30.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:30.082+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:30.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:30.083+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:33:30.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:30.737+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:30.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:30.737+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:30.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:30.737+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:30.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:30.738+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:33:30.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:30.696+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:30.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:30.696+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:30.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:30.696+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:30.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:30.696+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:33:31.375 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:33:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:33:31.248+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 14134 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:33:31.376 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:31.099+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:31.376 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:31.099+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:31.376 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:31.099+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:31.376 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:31.100+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:33:32.133 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:31.723+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:32.134 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:31.723+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:32.134 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:31.723+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:32.134 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:31.724+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:33:32.134 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:31.735+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:32.134 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:31.735+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:32.134 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:31.735+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:32.135 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:31.736+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:33:32.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:32.132+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:32.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:32.132+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:32.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:32.132+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:32.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:32.133+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:33:33.175 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:32.733+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:33.175 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:32.733+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:33.175 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:32.733+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:33.175 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:32.734+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:33:33.176 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:32.707+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:33.176 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:32.707+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:33.176 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:32.707+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:33.176 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:32.708+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:33:33.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:33.173+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:33.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:33.173+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:33.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:33.173+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:33.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:33.174+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:33:33.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:33.659+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:33.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:33.659+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:33.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:33.659+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:33.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:33.660+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:33:33.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:33.708+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:33.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:33.708+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:33.949 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:33.708+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:33.949 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:33.709+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:33:34.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:34.194+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:34.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:34.194+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:34.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:34.194+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:34.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:34.195+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:33:34.734 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:34.648+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:34.734 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:34.648+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:34.735 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:34.648+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:34.735 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:34.649+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:33:35.167 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:34.732+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:35.168 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:34.732+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:35.168 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:34.732+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:35.168 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:34.734+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:33:35.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:35.166+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:35.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:35.166+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:35.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:35.166+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:35.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:35.167+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:33:35.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:35.650+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:35.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:35.650+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:35.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:35.650+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:35.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:35.651+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:33:35.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:35.768+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:35.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:35.768+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:35.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:35.768+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:35.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:35.769+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:33:36.447 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:33:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:33:36.250+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 14231 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:33:36.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:36.138+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:36.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:36.138+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:36.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:36.138+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:36.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:36.139+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:33:36.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:36.677+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:36.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:36.677+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:36.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:36.677+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:36.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:36.677+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:33:36.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:36.771+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:36.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:36.771+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:36.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:36.771+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:36.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:36.771+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:33:37.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:37.099+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:37.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:37.099+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:37.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:37.099+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:37.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:37.099+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:33:37.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:37.672+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:37.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:37.672+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:37.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:37.672+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:37.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:37.673+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:33:37.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:37.797+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:37.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:37.797+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:37.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:37.797+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:37.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:37.797+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:33:38.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:38.088+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:38.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:38.088+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:38.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:38.088+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:38.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:38.089+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:33:39.079 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:38.706+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:39.080 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:38.706+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:39.080 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:38.706+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:39.080 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:38.707+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:33:39.080 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:38.833+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:39.080 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:38.833+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:39.081 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:38.833+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:39.081 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:38.833+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:33:39.335 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:39.079+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:39.335 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:39.079+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:39.335 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:39.079+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:39.335 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:39.080+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:33:40.058 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:39.724+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:40.058 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:39.724+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:40.058 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:39.724+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:40.059 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:39.725+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:33:40.059 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:39.792+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:40.059 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:39.792+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:40.059 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:39.792+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:40.059 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:39.793+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:33:40.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:40.057+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:40.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:40.057+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:40.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:40.057+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:40.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:40.059+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:33:41.079 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:40.774+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:41.080 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:40.774+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:41.080 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:40.774+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:41.080 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:40.775+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:33:41.080 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:40.719+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:41.080 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:40.719+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:41.080 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:40.719+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:41.081 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:40.719+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:33:41.376 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:33:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:33:41.253+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 14325 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:33:41.376 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:41.079+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:41.376 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:41.079+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:41.376 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:41.079+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:41.377 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:41.080+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:33:42.052 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:41.767+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:42.052 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:41.767+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:42.052 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:41.767+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:42.053 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:41.768+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:33:42.053 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:41.708+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:42.053 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:41.708+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:42.053 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:41.708+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:42.053 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:41.709+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:33:42.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:42.052+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:42.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:42.052+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:42.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:42.052+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:42.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:42.052+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:33:42.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:42.808+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:42.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:42.808+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:42.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:42.808+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:42.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:42.808+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:33:42.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:42.660+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:42.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:42.660+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:42.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:42.660+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:42.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:42.661+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:33:43.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:43.082+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:43.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:43.082+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:43.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:43.082+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:43.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:43.082+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:33:43.860 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:43.689+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:43.860 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:43.689+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:43.860 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:43.689+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:43.860 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:43.690+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:33:44.111 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:43.858+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:44.111 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:43.858+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:44.111 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:43.858+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:44.111 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:43.859+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:33:44.380 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:44.109+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:44.380 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:44.109+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:44.380 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:44.109+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:44.380 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:44.110+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:33:44.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:44.665+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:44.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:44.665+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:44.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:44.665+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:44.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:44.666+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:33:44.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:44.842+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:44.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:44.842+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:44.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:44.842+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:44.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:44.842+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:33:45.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:45.158+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:45.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:45.158+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:45.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:45.158+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:45.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:45.158+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:33:45.979 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:45.697+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:45.980 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:45.697+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:45.980 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:45.697+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:45.980 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:45.697+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:33:45.980 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:45.804+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:45.980 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:45.804+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:45.981 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:45.804+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:45.981 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:45.805+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:33:46.255 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:46.192+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:46.255 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:46.192+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:46.255 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:46.192+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:46.256 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:46.193+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:33:46.697 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:33:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:33:46.253+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 14422 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:33:47.156 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:46.798+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:47.156 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:46.798+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:47.157 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:46.798+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:47.157 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:46.798+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:33:47.157 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:46.716+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:47.157 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:46.716+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:47.157 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:46.716+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:47.157 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:46.717+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:33:47.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:47.155+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:47.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:47.155+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:47.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:47.155+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:47.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:47.156+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:33:48.171 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:47.755+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:48.171 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:47.755+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:48.172 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:47.755+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:48.172 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:47.756+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:33:48.172 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:47.771+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:48.172 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:47.771+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:48.172 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:47.771+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:48.172 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:47.772+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:33:48.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:48.169+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:48.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:48.169+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:48.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:48.169+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:48.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:48.170+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:33:49.147 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:48.789+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:49.147 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:48.789+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:49.148 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:48.789+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:49.148 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:48.790+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:33:49.148 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:48.749+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:49.148 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:48.749+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:49.148 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:48.749+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:49.148 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:48.750+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:33:49.424 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:49.145+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:49.424 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:49.145+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:49.424 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:49.145+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:49.424 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:49.147+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:33:50.117 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:49.766+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:50.118 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:49.766+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:50.118 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:49.766+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:50.118 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:49.767+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:33:50.118 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:49.780+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:50.118 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:49.780+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:50.118 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:49.780+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:50.119 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:49.781+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:33:50.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:50.116+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:50.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:50.116+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:50.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:50.116+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:50.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:50.117+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:33:51.075 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:50.789+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:51.075 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:50.789+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:51.075 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:50.789+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:51.076 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:50.790+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:33:51.076 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:50.792+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:51.076 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:50.792+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:51.076 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:50.792+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:51.076 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:50.793+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:33:51.381 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:33:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:33:51.255+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 14516 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:33:51.381 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:51.073+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:51.381 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:51.073+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:51.381 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:51.073+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:51.381 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:51.075+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:33:52.058 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:51.813+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:52.058 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:51.813+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:52.058 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:51.813+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:52.058 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:51.814+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:33:52.059 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:51.745+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:52.059 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:51.745+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:52.059 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:51.745+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:52.059 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:51.745+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:33:52.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:52.056+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:52.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:52.056+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:52.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:52.056+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:52.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:52.057+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:33:53.043 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:52.699+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:53.044 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:52.699+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:53.044 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:52.699+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:53.044 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:52.700+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:33:53.044 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:52.790+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:53.044 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:52.790+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:53.045 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:52.790+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:53.045 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:52.791+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:33:53.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:53.042+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:53.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:53.042+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:53.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:53.042+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:53.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:53.043+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:33:53.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:53.728+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:53.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:53.728+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:53.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:53.728+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:53.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:53.729+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:33:53.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:53.807+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:53.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:53.807+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:53.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:53.807+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:53.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:53.808+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:33:54.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:54.015+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:54.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:54.015+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:54.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:54.015+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:54.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:54.015+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:33:54.828 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:54.716+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:54.828 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:54.716+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:54.828 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:54.716+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:54.828 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:54.717+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:33:55.196 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:54.826+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:55.196 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:54.826+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:55.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:54.826+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:55.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:54.827+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:33:55.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:55.038+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:55.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:55.038+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:55.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:55.038+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:55.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:55.039+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:33:55.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:55.830+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:55.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:55.830+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:55.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:55.830+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:55.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:55.831+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:33:55.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:55.690+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:55.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:55.690+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:55.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:55.690+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:55.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:55.691+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:33:56.383 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:33:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:33:56.257+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 14607 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:33:56.384 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:56.045+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:56.384 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:56.045+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:56.384 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:56.045+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:56.384 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:56.046+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:33:57.002 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:56.836+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:57.002 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:56.836+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:57.002 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:56.836+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:57.002 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:56.837+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:33:57.002 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:56.729+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:57.003 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:56.729+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:57.003 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:56.729+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:57.003 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:56.730+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:33:57.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:57.000+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:57.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:57.000+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:57.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:57.000+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:57.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:57.001+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:33:58.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:57.768+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:58.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:57.768+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:58.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:57.768+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:58.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:57.769+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:33:58.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:57.883+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:58.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:57.883+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:58.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:57.883+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:58.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:57.884+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:33:58.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:57.991+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:58.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:57.991+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:58.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:57.991+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:58.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:57.992+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:33:59.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:58.968+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:59.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:58.968+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:59.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:58.968+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:33:59.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:58.969+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:33:59.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:58.850+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:59.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:58.850+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:59.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:58.850+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:33:59.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:58.851+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:33:59.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:58.819+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:59.199 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:58.819+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:59.199 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:58.819+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:33:59.199 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:58.820+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:34:00.153 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:59.994+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:00.153 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:59.994+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:00.153 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:59.994+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:00.153 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:33:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:33:59.995+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:34:00.154 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:59.804+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:00.154 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:59.804+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:00.154 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:59.804+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:00.154 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:33:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:33:59.805+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:34:00.154 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:59.805+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:00.155 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:59.805+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:00.155 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:59.805+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:00.155 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:33:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:33:59.805+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:34:01.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:00.765+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:01.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:00.765+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:01.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:00.765+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:01.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:00.766+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:34:01.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:00.769+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:01.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:00.769+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:01.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:00.769+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:01.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:00.770+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:34:01.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:00.963+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:01.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:00.963+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:01.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:00.963+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:01.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:00.965+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:34:01.697 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:34:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:34:01.258+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 14695 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:34:02.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:01.806+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:02.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:01.806+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:02.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:01.806+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:02.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:01.807+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:34:02.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:01.793+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:02.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:01.793+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:02.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:01.793+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:02.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:01.794+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:34:02.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:01.995+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:02.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:01.995+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:02.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:01.995+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:02.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:01.996+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:34:03.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:02.749+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:03.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:02.749+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:03.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:02.749+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:03.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:02.750+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:34:03.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:02.854+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:03.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:02.854+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:03.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:02.854+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:03.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:02.854+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:34:03.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:02.989+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:03.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:02.989+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:03.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:02.989+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:03.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:02.990+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:34:03.859 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:03.787+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:03.859 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:03.787+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:03.860 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:03.787+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:03.860 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:03.788+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:34:04.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:03.959+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:04.329 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:03.959+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:04.329 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:03.959+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:04.329 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:03.960+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:34:04.329 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:03.857+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:04.329 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:03.857+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:04.329 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:03.857+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:04.330 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:03.858+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:34:04.873 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:04.783+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:04.874 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:04.783+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:04.874 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:04.783+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:04.874 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:04.784+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:34:05.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:04.872+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:05.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:04.872+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:05.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:04.872+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:05.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:04.873+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:34:05.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:04.972+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:05.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:04.972+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:05.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:04.972+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:05.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:04.973+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:34:06.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:05.860+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:06.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:05.860+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:06.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:05.860+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:06.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:05.861+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:34:06.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:05.740+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:06.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:05.740+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:06.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:05.740+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:06.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:05.740+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:34:06.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:05.956+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:06.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:05.956+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:06.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:05.956+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:06.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:05.956+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:34:06.697 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:34:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:34:06.260+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 14785 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:34:07.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:06.880+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:07.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:06.880+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:07.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:06.880+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:07.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:06.881+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:34:07.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:06.727+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:07.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:06.727+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:07.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:06.727+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:07.199 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:06.727+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:34:07.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:06.932+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:07.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:06.932+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:07.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:06.932+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:07.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:06.933+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:34:08.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:07.861+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:08.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:07.861+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:08.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:07.861+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:08.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:07.862+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:34:08.199 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:07.708+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:08.199 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:07.708+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:08.199 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:07.708+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:08.200 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:07.709+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:34:08.200 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:07.895+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:08.200 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:07.895+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:08.201 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:07.895+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:08.201 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:07.895+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:34:08.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:08.867+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:08.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:08.867+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:08.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:08.867+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:08.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:08.868+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:34:08.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:08.680+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:08.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:08.680+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:08.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:08.680+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:08.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:08.680+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:34:08.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:08.853+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:08.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:08.853+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:08.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:08.853+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:08.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:08.854+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:34:09.871 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:09.682+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:09.871 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:09.682+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:09.871 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:09.682+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:09.871 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:09.682+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:34:10.154 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:09.901+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:10.154 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:09.901+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:10.154 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:09.901+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:10.154 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:09.902+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:34:10.155 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:09.869+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:10.155 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:09.869+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:10.155 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:09.869+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:10.155 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:09.870+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:34:10.833 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:10.697+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:10.834 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:10.697+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:10.834 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:10.697+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:10.834 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:10.698+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:34:11.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:10.892+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:11.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:10.892+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:11.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:10.892+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:11.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:10.893+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:34:11.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:10.830+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:11.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:10.830+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:11.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:10.830+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:11.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:10.831+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:34:11.697 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:34:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:34:11.261+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 14873 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:34:12.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:11.940+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:12.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:11.940+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:12.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:11.940+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:12.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:11.941+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:34:12.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:11.702+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:12.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:11.702+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:12.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:11.702+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:12.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:11.703+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:34:12.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:11.789+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:12.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:11.789+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:12.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:11.789+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:12.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:11.790+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:34:13.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:12.733+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:13.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:12.733+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:13.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:12.733+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:13.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:12.734+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:34:13.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:12.898+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:13.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:12.898+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:13.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:12.898+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:13.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:12.899+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:34:13.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:12.752+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:13.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:12.752+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:13.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:12.752+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:13.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:12.753+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:34:13.925 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:13.728+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:13.926 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:13.728+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:13.926 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:13.728+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:13.926 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:13.729+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:34:13.926 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:13.802+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:13.926 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:13.802+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:13.926 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:13.802+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:13.927 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:13.803+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:34:14.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:13.924+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:14.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:13.924+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:14.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:13.924+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:14.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:13.925+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:34:14.916 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:14.826+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:14.916 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:14.826+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:14.917 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:14.826+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:14.917 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:14.826+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:34:14.917 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:14.698+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:14.917 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:14.698+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:14.917 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:14.698+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:14.918 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:14.699+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:34:15.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:14.915+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:15.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:14.915+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:15.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:14.915+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:15.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:14.916+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:34:16.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:15.881+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:16.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:15.881+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:16.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:15.881+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:16.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:15.882+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:34:16.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:15.742+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:16.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:15.742+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:16.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:15.742+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:16.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:15.743+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:34:16.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:15.847+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:16.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:15.848+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:16.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:15.848+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:16.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:15.849+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:34:16.697 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:34:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:34:16.263+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 14964 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:34:17.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:16.914+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:17.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:16.914+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:17.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:16.914+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:17.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:16.915+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:34:17.199 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:16.705+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:17.199 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:16.705+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:17.199 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:16.705+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:17.199 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:16.706+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:34:17.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:16.810+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:17.200 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:16.810+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:17.200 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:16.810+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:17.200 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:16.810+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:34:17.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:17.916+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:17.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:17.916+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:17.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:17.916+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:17.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:17.917+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:34:17.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:17.678+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:17.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:17.678+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:17.949 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:17.678+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:17.949 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:17.679+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:34:17.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:17.843+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:17.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:17.843+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:17.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:17.843+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:17.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:17.844+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:34:18.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:18.882+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:18.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:18.882+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:18.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:18.882+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:18.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:18.883+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:34:18.949 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:18.658+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:18.949 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:18.658+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:18.949 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:18.658+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:18.950 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:18.659+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:34:18.950 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:18.864+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:18.950 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:18.864+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:18.951 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:18.864+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:18.951 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:18.865+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:34:19.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:19.880+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:19.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:19.880+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:19.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:19.880+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:19.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:19.881+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:34:19.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:19.678+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:19.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:19.678+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:19.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:19.678+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:19.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:19.679+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:34:19.949 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:19.846+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:19.949 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:19.846+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:19.949 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:19.846+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:19.949 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:19.847+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:34:20.911 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:20.828+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:20.911 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:20.828+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:20.911 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:20.828+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:20.912 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:20.829+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:34:20.912 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:20.653+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:20.912 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:20.653+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:20.912 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:20.653+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:20.912 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:20.654+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:34:21.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:20.909+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:21.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:20.909+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:21.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:20.909+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:21.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:20.910+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:34:21.611 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:34:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:34:21.265+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 15052 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:34:21.940 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:21.610+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:21.940 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:21.610+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:21.940 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:21.610+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:21.940 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:21.611+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:34:21.941 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:21.794+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:21.941 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:21.794+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:21.941 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:21.794+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:21.941 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:21.795+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:34:22.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:21.938+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:22.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:21.938+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:22.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:21.938+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:22.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:21.939+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:34:22.899 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:22.640+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:22.899 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:22.640+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:22.900 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:22.640+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:22.900 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:22.641+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:34:22.900 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:22.769+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:22.900 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:22.769+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:22.900 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:22.769+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:22.900 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:22.770+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:34:23.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:22.897+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:23.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:22.897+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:23.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:22.898+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:23.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:22.899+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:34:23.911 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:23.752+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:23.911 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:23.752+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:23.911 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:23.752+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:23.911 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:23.753+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:34:23.911 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:23.617+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:23.911 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:23.617+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:23.912 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:23.617+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:23.912 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:23.618+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:34:24.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:23.909+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:24.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:23.909+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:24.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:23.909+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:24.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:23.910+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:34:24.870 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:24.707+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:24.871 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:24.707+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:24.871 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:24.707+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:24.871 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:24.708+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:34:24.871 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:24.601+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:24.871 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:24.601+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:24.871 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:24.601+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:24.872 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:24.602+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:34:25.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:24.870+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:25.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:24.870+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:25.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:24.870+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:25.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:24.871+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:34:25.915 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:25.865+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:25.916 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:25.865+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:25.916 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:25.865+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:25.916 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:25.866+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:34:25.916 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:25.756+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:25.916 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:25.756+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:25.916 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:25.756+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:25.917 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:25.757+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:34:25.917 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:25.643+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:25.917 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:25.643+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:25.917 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:25.643+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:25.917 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:25.644+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:34:26.617 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:34:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:34:26.267+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 15142 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:34:26.889 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:26.758+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:26.889 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:26.758+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:26.889 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:26.758+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:26.889 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:26.759+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:34:26.890 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:26.617+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:26.890 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:26.617+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:26.890 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:26.617+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:26.890 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:26.618+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:34:27.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:26.888+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:27.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:26.888+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:27.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:26.888+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:27.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:26.889+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:34:27.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:27.648+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:27.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:27.648+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:27.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:27.648+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:27.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:27.649+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:34:27.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:27.851+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:27.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:27.851+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:27.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:27.851+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:27.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:27.852+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:34:27.949 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:27.739+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:27.949 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:27.739+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:27.949 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:27.739+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:27.949 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:27.739+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:34:29.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:28.763+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:29.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:28.763+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:29.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:28.763+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:29.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:28.764+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:34:29.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:28.696+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:29.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:28.696+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:29.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:28.696+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:29.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:28.697+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:34:29.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:28.841+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:29.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:28.841+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:29.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:28.841+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:29.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:28.842+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:34:30.155 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:29.846+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:30.155 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:29.846+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:30.155 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:29.846+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:30.155 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:29.847+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:34:30.156 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:29.776+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:30.156 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:29.776+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:30.156 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:29.776+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:30.156 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:29.776+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:34:30.156 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:29.736+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:30.157 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:29.736+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:30.157 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:29.736+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:30.157 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:29.737+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:34:31.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:30.786+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:31.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:30.786+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:31.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:30.786+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:31.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:30.787+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:34:31.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:30.704+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:31.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:30.704+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:31.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:30.704+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:31.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:30.705+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:34:31.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:30.841+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:31.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:30.841+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:31.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:30.841+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:31.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:30.842+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:34:31.682 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:34:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:34:31.268+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 15230 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:34:31.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:31.799+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:31.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:31.799+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:31.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:31.799+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:31.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:31.800+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:34:31.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:31.863+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:31.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:31.863+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:31.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:31.863+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:31.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:31.864+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:34:31.949 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:31.681+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:31.950 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:31.681+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:31.950 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:31.681+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:31.950 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:31.682+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:34:32.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:32.828+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:32.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:32.828+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:32.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:32.828+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:32.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:32.829+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:34:32.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:32.861+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:32.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:32.861+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:32.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:32.861+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:32.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:32.862+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:34:32.949 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:32.688+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:32.949 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:32.688+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:32.949 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:32.688+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:32.949 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:32.689+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:34:33.904 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:33.665+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:33.905 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:33.665+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:33.905 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:33.665+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:33.905 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:33.666+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:34:33.905 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:33.795+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:33.905 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:33.795+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:33.905 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:33.795+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:33.906 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:33.796+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:34:34.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:33.903+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:34.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:33.903+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:34.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:33.903+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:34.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:33.904+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:34:35.131 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:34.701+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:35.132 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:34.701+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:35.132 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:34.701+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:35.132 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:34.702+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:34:35.132 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:34.799+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:35.133 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:34.799+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:35.133 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:34.799+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:35.133 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:34.800+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:34:35.133 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:34.916+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:35.133 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:34.916+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:35.133 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:34.916+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:35.134 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:34.918+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:34:35.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:35.693+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:35.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:35.693+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:35.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:35.693+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:35.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:35.694+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:34:35.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:35.800+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:35.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:35.800+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:35.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:35.800+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:35.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:35.801+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:34:35.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:35.896+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:35.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:35.896+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:35.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:35.896+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:35.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:35.897+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:34:36.671 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:34:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:34:36.270+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 15321 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:34:36.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:36.911+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:36.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:36.911+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:36.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:36.911+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:36.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:36.912+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:34:36.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:36.776+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:36.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:36.776+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:36.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:36.776+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:36.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:36.777+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:34:36.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:36.669+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:36.949 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:36.669+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:36.949 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:36.669+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:36.949 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:36.670+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:34:37.914 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:37.657+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:37.914 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:37.657+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:37.915 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:37.657+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:37.915 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:37.658+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:34:37.915 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:37.742+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:37.915 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:37.742+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:37.915 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:37.742+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:37.915 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:37.743+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:34:38.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:37.913+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:38.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:37.913+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:38.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:37.913+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:38.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:37.914+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:34:39.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:38.943+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:39.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:38.943+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:39.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:38.943+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:39.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:38.944+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:34:39.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:38.742+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:39.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:38.742+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:39.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:38.742+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:39.199 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:38.743+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:34:39.199 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:38.698+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:39.199 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:38.698+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:39.199 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:38.698+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:39.200 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:38.699+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:34:39.974 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:39.707+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:39.975 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:39.707+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:39.975 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:39.707+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:39.975 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:39.708+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:34:39.975 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:39.746+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:39.975 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:39.746+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:39.975 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:39.746+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:39.976 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:39.747+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:34:40.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:39.973+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:40.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:39.973+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:40.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:39.973+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:40.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:39.974+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:34:41.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:40.976+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:41.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:40.976+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:41.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:40.976+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:41.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:40.976+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:34:41.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:40.740+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:41.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:40.740+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:41.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:40.740+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:41.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:40.740+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:34:41.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:40.745+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:41.199 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:40.745+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:41.199 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:40.745+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:41.199 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:40.746+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:34:41.698 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:34:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:34:41.271+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 15408 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:34:42.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:41.956+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:42.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:41.956+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:42.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:41.956+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:42.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:41.956+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:34:42.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:41.790+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:42.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:41.790+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:42.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:41.790+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:42.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:41.791+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:34:42.199 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:41.732+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:42.199 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:41.732+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:42.199 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:41.732+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:42.199 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:41.733+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:34:43.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:42.993+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:43.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:42.993+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:43.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:42.993+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:43.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:42.994+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:34:43.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:42.829+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:43.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:42.829+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:43.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:42.829+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:43.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:42.830+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:34:43.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:42.772+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:43.199 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:42.772+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:43.199 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:42.772+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:43.199 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:42.773+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:34:43.835 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:43.738+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:43.835 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:43.738+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:43.835 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:43.738+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:43.835 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:43.739+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:34:44.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:43.833+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:44.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:43.833+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:44.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:43.833+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:44.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:43.834+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:34:44.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:43.947+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:44.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:43.947+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:44.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:43.947+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:44.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:43.948+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:34:45.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:44.839+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:45.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:44.839+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:45.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:44.839+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:45.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:44.840+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:34:45.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:44.726+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:45.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:44.726+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:45.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:44.726+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:45.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:44.727+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:34:45.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:44.946+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:45.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:44.946+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:45.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:44.946+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:45.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:44.947+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:34:46.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:45.888+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:46.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:45.888+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:46.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:45.888+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:46.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:45.889+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:34:46.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:45.697+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:46.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:45.697+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:46.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:45.697+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:46.202 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:45.698+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:34:46.202 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:45.921+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:46.202 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:45.921+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:46.202 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:45.921+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:46.202 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:45.922+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:34:46.666 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:34:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:34:46.273+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 15498 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:34:46.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:46.870+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:46.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:46.870+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:46.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:46.870+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:46.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:46.871+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:34:46.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:46.665+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:46.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:46.665+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:46.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:46.665+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:46.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:46.666+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:34:47.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:46.972+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:47.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:46.972+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:47.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:46.972+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:47.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:46.973+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:34:47.911 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:47.618+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:47.911 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:47.618+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:47.911 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:47.618+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:47.911 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:47.619+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:34:48.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:47.909+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:48.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:47.909+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:48.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:47.909+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:48.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:47.910+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:34:48.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:47.932+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:48.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:47.932+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:48.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:47.932+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:48.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:47.933+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:34:48.900 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:48.624+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:48.900 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:48.624+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:48.900 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:48.624+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:48.900 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:48.625+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:34:49.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:48.898+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:49.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:48.898+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:49.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:48.898+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:49.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:48.899+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:34:49.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:48.903+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:49.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:48.903+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:49.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:48.903+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:49.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:48.904+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:34:49.945 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:49.659+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:49.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:49.659+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:49.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:49.659+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:49.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:49.660+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:34:49.946 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:49.908+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:49.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:49.908+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:49.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:49.908+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:49.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:49.909+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:34:50.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:49.945+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:50.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:49.945+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:50.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:49.945+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:50.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:49.946+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:34:50.887 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:50.644+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:50.887 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:50.644+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:50.887 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:50.644+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:50.887 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:50.645+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:34:51.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:50.978+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:51.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:50.978+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:51.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:50.978+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:51.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:50.979+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:34:51.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:50.885+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:51.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:50.885+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:51.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:50.885+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:51.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:50.886+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:34:51.691 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:34:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:34:51.275+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 15586 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:34:51.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:51.897+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:51.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:51.897+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:51.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:51.897+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:51.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:51.898+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:34:51.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:51.689+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:51.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:51.689+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:51.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:51.689+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:51.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:51.690+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:34:52.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:52.017+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:52.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:52.017+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:52.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:52.017+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:52.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:52.018+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:34:52.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:52.919+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:52.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:52.919+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:52.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:52.919+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:52.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:52.920+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:34:52.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:52.682+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:52.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:52.682+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:52.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:52.682+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:52.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:52.684+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:34:53.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:53.026+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:53.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:53.026+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:53.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:53.026+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:53.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:53.027+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:34:53.888 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:53.661+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:53.888 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:53.661+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:53.889 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:53.661+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:53.889 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:53.662+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:34:54.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:53.887+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:54.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:53.887+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:54.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:53.887+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:54.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:53.888+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:34:54.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:54.051+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:54.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:54.051+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:54.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:54.051+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:54.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:54.052+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:34:54.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:54.852+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:54.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:54.852+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:54.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:54.852+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:54.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:54.852+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:34:54.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:54.658+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:54.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:54.658+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:54.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:54.658+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:54.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:54.659+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:34:55.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:55.086+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:55.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:55.086+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:55.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:55.086+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:55.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:55.087+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:34:55.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:55.880+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:55.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:55.880+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:55.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:55.880+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:55.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:55.881+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:34:55.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:55.655+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:55.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:55.655+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:55.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:55.655+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:55.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:55.656+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:34:56.403 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:34:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:34:56.276+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 15678 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:34:56.403 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:56.069+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:56.404 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:56.069+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:56.404 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:56.069+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:56.404 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:56.070+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:34:57.056 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:56.892+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:57.056 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:56.892+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:57.056 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:56.892+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:57.057 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:56.893+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:34:57.057 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:56.698+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:57.057 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:56.698+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:57.057 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:56.698+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:57.057 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:56.699+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:34:57.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:57.054+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:57.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:57.054+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:57.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:57.054+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:57.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:57.055+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:34:57.940 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:57.662+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:57.940 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:57.662+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:57.940 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:57.662+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:57.941 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:57.662+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:34:58.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:57.938+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:58.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:57.938+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:58.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:57.938+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:58.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:57.940+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:34:58.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:58.007+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:58.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:58.007+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:58.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:58.007+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:58.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:58.008+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:34:58.944 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:58.649+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:58.944 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:58.649+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:58.944 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:58.649+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:58.944 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:58.650+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:34:59.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:58.942+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:59.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:58.942+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:59.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:58.942+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:34:59.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:58.943+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:34:59.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:59.021+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:59.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:59.021+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:59.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:59.021+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:34:59.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:34:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:34:59.022+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:34:59.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:59.612+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:59.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:59.612+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:59.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:59.612+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:34:59.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:34:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:34:59.613+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:35:00.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:00.032+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:00.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:00.032+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:00.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:00.032+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:00.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:00.032+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:35:00.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:59.974+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:00.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:59.974+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:00.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:59.974+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:00.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:34:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:34:59.974+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:35:00.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:00.646+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:00.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:00.646+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:00.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:00.646+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:00.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:00.647+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:35:01.280 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:00.986+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:01.280 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:00.986+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:01.280 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:00.986+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:01.281 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:00.986+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:35:01.281 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:01.074+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:01.281 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:01.074+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:01.281 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:01.074+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:01.281 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:01.075+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:35:01.661 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:35:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:35:01.278+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 15767 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:35:01.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:01.659+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:01.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:01.659+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:01.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:01.659+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:01.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:01.660+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:35:02.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:02.004+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:02.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:02.004+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:02.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:02.004+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:02.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:02.004+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:35:02.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:02.092+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:02.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:02.092+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:02.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:02.092+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:02.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:02.093+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:35:02.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:02.621+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:02.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:02.621+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:02.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:02.621+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:02.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:02.622+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:35:03.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:03.041+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:03.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:03.041+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:03.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:03.041+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:03.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:03.042+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:35:03.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:03.076+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:03.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:03.076+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:03.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:03.076+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:03.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:03.077+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:35:03.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:03.654+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:03.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:03.654+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:03.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:03.654+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:03.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:03.655+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:35:04.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:04.125+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:04.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:04.125+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:04.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:04.125+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:04.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:04.126+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:35:04.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:04.046+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:04.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:04.046+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:04.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:04.046+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:04.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:04.046+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:35:04.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:04.695+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:04.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:04.695+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:04.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:04.695+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:04.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:04.696+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:35:05.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:05.084+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:05.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:05.084+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:05.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:05.084+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:05.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:05.085+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:35:05.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:05.153+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:05.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:05.153+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:05.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:05.153+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:05.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:05.154+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:35:05.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:05.694+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:05.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:05.694+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:05.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:05.694+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:05.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:05.695+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:35:06.407 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:06.111+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:06.442 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:06.111+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:06.443 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:06.111+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:06.443 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:06.112+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:35:06.443 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:06.117+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:06.443 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:06.117+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:06.443 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:06.117+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:06.444 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:06.118+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:35:06.444 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:35:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:35:06.280+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 15857 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:35:07.104 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:06.740+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:07.104 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:06.740+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:07.104 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:06.740+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:07.105 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:06.741+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:35:07.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:07.130+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:07.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:07.130+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:07.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:07.130+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:07.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:07.130+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:35:07.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:07.103+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:07.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:07.103+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:07.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:07.103+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:07.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:07.104+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:35:08.089 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:07.747+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:08.089 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:07.747+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:08.089 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:07.747+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:08.089 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:07.748+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:35:08.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:08.102+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:08.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:08.102+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:08.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:08.102+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:08.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:08.103+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:35:08.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:08.087+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:08.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:08.087+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:08.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:08.087+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:08.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:08.088+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:35:09.066 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:08.783+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:09.066 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:08.783+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:09.066 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:08.783+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:09.067 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:08.783+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:35:09.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:09.066+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:09.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:09.066+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:09.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:09.066+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:09.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:09.066+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:35:09.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:09.144+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:09.449 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:09.144+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:09.449 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:09.144+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:09.449 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:09.144+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:35:10.052 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:09.740+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:10.052 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:09.740+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:10.052 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:09.740+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:10.052 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:09.741+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:35:10.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:10.051+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:10.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:10.051+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:10.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:10.051+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:10.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:10.052+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:35:10.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:10.113+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:10.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:10.113+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:10.449 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:10.113+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:10.449 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:10.114+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:35:11.077 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:10.785+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:11.077 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:10.785+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:11.077 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:10.785+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:11.077 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:10.786+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:35:11.406 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:35:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:35:11.282+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 15946 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:35:11.406 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:11.163+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:11.407 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:11.163+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:11.407 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:11.163+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:11.407 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:11.164+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:35:11.407 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:11.076+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:11.408 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:11.076+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:11.408 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:11.076+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:11.408 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:11.077+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:35:12.100 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:11.821+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:12.100 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:11.821+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:12.101 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:11.821+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:12.101 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:11.821+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:35:12.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:12.160+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:12.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:12.160+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:12.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:12.160+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:12.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:12.160+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:35:12.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:12.100+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:12.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:12.100+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:12.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:12.100+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:12.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:12.101+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:35:13.094 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:12.796+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:13.094 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:12.796+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:13.094 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:12.796+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:13.094 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:12.796+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:35:13.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:13.093+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:13.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:13.093+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:13.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:13.093+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:13.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:13.094+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:35:13.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:13.122+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:13.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:13.122+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:13.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:13.122+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:13.449 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:13.123+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:35:13.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:13.780+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:13.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:13.780+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:13.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:13.780+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:13.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:13.781+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:35:14.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:14.103+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:14.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:14.103+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:14.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:14.103+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:14.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:14.104+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:35:14.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:14.068+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:14.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:14.068+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:14.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:14.068+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:14.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:14.069+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:35:15.067 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:14.777+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:15.067 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:14.777+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:15.068 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:14.777+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:15.068 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:14.777+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:35:15.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:15.081+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:15.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:15.081+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:15.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:15.081+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:15.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:15.081+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:35:15.449 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:15.115+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:15.449 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:15.115+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:15.449 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:15.115+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:15.449 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:15.116+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:35:16.091 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:15.743+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:16.092 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:15.743+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:16.092 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:15.743+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:16.092 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:15.745+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:35:16.409 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:16.090+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:16.410 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:16.090+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:16.410 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:16.090+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:16.410 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:16.091+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:35:16.410 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:35:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:35:16.284+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 16038 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:35:16.411 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:16.102+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:16.411 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:16.102+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:16.411 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:16.102+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:16.411 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:16.102+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:35:17.055 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:16.699+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:17.055 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:16.699+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:17.056 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:16.699+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:17.056 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:16.699+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:35:17.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:17.054+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:17.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:17.054+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:17.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:17.054+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:17.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:17.055+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:35:17.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:17.147+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:17.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:17.147+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:17.449 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:17.147+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:17.449 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:17.148+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:35:18.056 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:17.705+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:18.057 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:17.705+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:18.057 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:17.705+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:18.057 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:17.706+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:35:18.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:18.181+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:18.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:18.181+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:18.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:18.181+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:18.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:18.182+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:35:18.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:18.055+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:18.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:18.055+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:18.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:18.055+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:18.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:18.056+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:35:19.039 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:18.702+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:19.040 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:18.702+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:19.040 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:18.702+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:19.040 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:18.703+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:35:19.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:19.152+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:19.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:19.152+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:19.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:19.152+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:19.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:19.153+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:35:19.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:19.038+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:19.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:19.038+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:19.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:19.038+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:19.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:19.039+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:35:19.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:19.689+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:19.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:19.689+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:19.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:19.689+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:19.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:19.689+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:35:20.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:20.009+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:20.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:20.009+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:20.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:20.009+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:20.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:20.010+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:35:20.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:20.127+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:20.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:20.127+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:20.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:20.127+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:20.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:20.128+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:35:21.015 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:20.731+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:21.015 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:20.731+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:21.015 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:20.731+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:21.016 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:20.731+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:35:21.287 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:21.013+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:21.287 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:21.013+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:21.287 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:21.013+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:21.287 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:21.014+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:35:21.288 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:21.135+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:21.288 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:21.135+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:21.288 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:21.135+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:21.288 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:21.136+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:35:21.697 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:35:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:35:21.285+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 16127 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:35:22.004 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:21.723+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:22.005 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:21.723+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:22.005 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:21.723+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:22.005 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:21.724+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:35:22.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:22.177+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:22.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:22.177+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:22.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:22.177+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:22.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:22.178+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:35:22.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:22.003+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:22.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:22.003+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:22.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:22.003+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:22.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:22.004+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:35:23.191 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:22.993+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:23.191 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:22.993+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:23.191 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:22.993+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:23.192 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:22.994+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:35:23.192 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:22.746+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:23.192 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:22.746+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:23.192 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:22.746+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:23.192 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:22.746+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:35:23.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:23.189+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:23.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:23.189+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:23.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:23.189+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:23.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:23.190+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:35:23.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:23.776+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:23.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:23.776+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:23.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:23.776+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:23.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:23.777+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:35:24.283 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:24.177+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:24.283 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:24.177+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:24.284 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:24.177+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:24.284 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:24.178+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:35:24.284 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:23.949+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:24.284 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:23.949+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:24.284 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:23.949+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:24.284 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:23.950+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:35:25.143 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:24.981+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:25.143 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:24.981+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:25.143 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:24.981+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:25.144 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:24.982+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:35:25.144 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:24.799+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:25.144 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:24.799+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:25.144 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:24.799+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:25.144 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:24.800+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:35:25.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:25.141+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:25.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:25.141+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:25.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:25.141+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:25.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:25.142+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:35:26.122 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:25.954+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:26.123 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:25.954+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:26.123 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:25.954+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:26.123 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:25.955+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:35:26.123 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:25.821+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:26.123 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:25.821+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:26.123 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:25.821+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:26.124 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:25.822+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:35:26.413 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:26.121+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:26.413 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:26.121+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:26.413 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:26.121+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:26.413 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:26.122+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:35:26.413 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:35:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:35:26.287+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 16217 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:35:27.110 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:26.803+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:27.111 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:26.803+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:27.111 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:26.803+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:27.111 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:26.804+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:35:27.111 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:26.979+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:27.111 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:26.979+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:27.111 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:26.979+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:27.112 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:26.980+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:35:27.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:27.109+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:27.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:27.109+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:27.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:27.109+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:27.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:27.110+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:35:28.114 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:27.758+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:28.115 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:27.758+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:28.115 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:27.758+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:28.115 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:27.759+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:35:28.115 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:27.935+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:28.115 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:27.935+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:28.115 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:27.935+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:28.116 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:27.936+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:35:28.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:28.113+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:28.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:28.113+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:28.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:28.113+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:28.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:28.114+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:35:29.089 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:28.805+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:29.089 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:28.805+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:29.089 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:28.805+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:29.089 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:28.806+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:35:29.090 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:28.905+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:29.090 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:28.905+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:29.090 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:28.905+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:29.090 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:28.906+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:35:29.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:29.087+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:29.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:29.087+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:29.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:29.087+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:29.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:29.088+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:35:30.078 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:29.890+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:30.079 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:29.890+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:30.079 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:29.890+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:30.079 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:29.890+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:35:30.080 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:29.775+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:30.080 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:29.775+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:30.080 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:29.775+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:30.080 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:29.776+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:35:30.429 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:30.076+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:30.430 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:30.076+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:30.430 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:30.076+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:30.430 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:30.077+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:35:31.079 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:30.775+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:31.079 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:30.775+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:31.079 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:30.775+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:31.079 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:30.776+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:35:31.080 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:30.860+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:31.080 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:30.860+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:31.080 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:30.860+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:31.080 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:30.861+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:35:31.412 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:35:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:35:31.289+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 16305 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:35:31.413 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:31.077+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:31.413 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:31.077+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:31.413 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:31.077+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:31.413 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:31.078+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:35:32.054 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:31.737+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:32.055 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:31.737+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:32.055 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:31.737+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:32.055 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:31.738+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:35:32.055 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:31.832+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:32.055 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:31.832+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:32.055 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:31.832+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:32.056 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:31.833+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:35:32.056 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:32.053+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:32.056 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:32.053+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:32.056 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:32.053+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:32.451 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:32.053+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:35:33.093 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:32.706+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:33.093 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:32.706+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:33.093 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:32.706+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:33.093 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:32.707+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:35:33.093 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:32.817+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:33.094 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:32.817+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:33.094 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:32.817+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:33.094 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:32.818+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:35:33.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:33.091+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:33.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:33.091+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:33.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:33.091+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:33.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:33.092+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:35:33.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:33.704+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:33.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:33.704+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:33.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:33.704+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:33.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:33.705+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:35:33.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:33.830+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:33.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:33.830+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:33.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:33.830+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:33.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:33.831+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:35:34.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:34.089+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:34.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:34.089+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:34.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:34.089+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:34.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:34.090+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:35:34.812 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:34.672+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:34.812 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:34.672+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:34.812 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:34.672+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:34.812 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:34.673+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:35:35.083 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:34.810+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:35.083 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:34.810+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:35.083 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:34.810+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:35.083 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:34.811+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:35:35.349 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:35.081+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:35.349 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:35.081+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:35.349 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:35.081+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:35.349 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:35.082+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:35:36.053 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:35.783+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:36.053 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:35.783+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:36.053 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:35.783+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:36.053 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:35.784+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:35:36.053 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:35.707+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:36.054 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:35.707+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:36.054 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:35.707+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:36.054 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:35.708+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:35:36.306 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:35:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:35:36.290+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 16396 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:35:36.306 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:36.051+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:36.306 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:36.051+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:36.306 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:36.051+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:36.307 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:36.052+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:35:36.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:36.735+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:36.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:36.735+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:36.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:36.735+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:36.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:36.736+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:35:36.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:36.746+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:36.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:36.746+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:36.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:36.746+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:36.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:36.747+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:35:37.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:37.058+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:37.545 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:37.058+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:37.546 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:37.058+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:37.546 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:37.059+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:35:38.070 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:37.724+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:38.071 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:37.724+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:38.071 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:37.724+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:38.071 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:37.724+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:35:38.071 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:37.782+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:38.071 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:37.782+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:38.072 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:37.782+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:38.072 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:37.783+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:35:38.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:38.069+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:38.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:38.069+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:38.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:38.069+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:38.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:38.070+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:35:39.060 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:38.737+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:39.061 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:38.737+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:39.061 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:38.737+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:39.061 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:38.738+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:35:39.061 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:38.741+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:39.061 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:38.741+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:39.061 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:38.741+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:39.062 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:38.742+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:35:39.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:39.059+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:39.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:39.059+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:39.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:39.059+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:39.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:39.060+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:35:39.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:39.775+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:39.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:39.775+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:39.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:39.775+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:39.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:39.776+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:35:39.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:39.694+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:39.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:39.694+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:39.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:39.694+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:39.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:39.695+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:35:40.394 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:40.063+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:40.395 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:40.063+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:40.395 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:40.063+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:40.395 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:40.064+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:35:40.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:40.763+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:40.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:40.763+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:40.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:40.763+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:40.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:40.764+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:35:40.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:40.668+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:40.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:40.668+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:40.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:40.668+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:40.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:40.669+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:35:41.306 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:35:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:35:41.292+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 16484 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:35:41.307 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:41.045+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:41.307 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:41.045+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:41.307 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:41.045+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:41.307 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:41.046+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:35:41.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:41.747+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:41.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:41.747+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:41.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:41.747+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:41.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:41.748+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:35:41.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:41.635+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:41.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:41.635+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:41.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:41.635+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:41.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:41.636+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:35:42.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:42.044+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:42.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:42.044+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:42.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:42.044+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:42.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:42.044+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:35:42.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:42.760+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:42.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:42.760+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:42.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:42.760+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:42.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:42.762+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:35:42.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:42.594+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:42.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:42.594+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:42.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:42.594+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:42.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:42.594+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:35:43.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:43.077+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:43.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:43.077+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:43.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:43.077+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:43.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:43.077+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:35:43.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:43.759+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:43.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:43.759+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:43.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:43.759+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:43.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:43.760+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:35:43.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:43.618+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:43.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:43.618+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:43.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:43.618+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:43.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:43.619+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:35:44.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:44.117+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:44.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:44.117+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:44.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:44.117+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:44.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:44.118+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:35:44.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:44.615+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:44.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:44.615+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:44.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:44.615+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:44.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:44.616+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:35:44.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:44.737+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:44.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:44.737+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:44.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:44.737+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:44.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:44.738+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:35:45.440 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:45.095+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:45.440 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:45.095+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:45.440 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:45.095+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:45.440 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:45.096+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:35:45.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:45.589+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:45.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:45.589+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:45.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:45.589+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:45.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:45.590+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:35:46.063 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:45.740+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:46.064 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:45.740+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:46.064 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:45.740+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:46.064 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:45.741+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:35:46.318 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:35:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:35:46.294+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 16574 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:35:46.318 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:46.062+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:46.318 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:46.062+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:46.319 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:46.062+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:46.319 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:46.063+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:35:46.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:46.691+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:46.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:46.691+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:46.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:46.691+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:46.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:46.692+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:35:46.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:46.584+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:46.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:46.584+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:46.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:46.584+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:46.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:46.585+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:35:47.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:47.098+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:47.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:47.098+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:47.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:47.098+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:47.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:47.099+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:35:47.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:47.609+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:47.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:47.609+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:47.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:47.609+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:47.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:47.610+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:35:47.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:47.679+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:47.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:47.679+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:47.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:47.679+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:47.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:47.680+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:35:48.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:48.125+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:48.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:48.125+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:48.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:48.125+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:48.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:48.125+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:35:48.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:48.665+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:48.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:48.665+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:48.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:48.665+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:48.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:48.666+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:35:48.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:48.598+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:48.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:48.598+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:48.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:48.598+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:48.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:48.598+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:35:49.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:49.121+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:49.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:49.121+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:49.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:49.121+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:49.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:49.122+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:35:49.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:49.699+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:49.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:49.699+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:49.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:49.699+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:49.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:49.700+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:35:49.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:49.618+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:49.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:49.618+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:49.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:49.618+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:49.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:49.619+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:35:50.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:50.130+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:50.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:50.130+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:50.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:50.130+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:50.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:50.131+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:35:50.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:50.640+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:50.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:50.640+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:50.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:50.640+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:50.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:50.640+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:35:50.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:50.667+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:50.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:50.667+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:50.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:50.667+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:50.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:50.667+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:35:51.422 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:35:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:35:51.296+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 16662 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:35:51.423 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:51.115+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:51.423 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:51.115+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:51.423 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:51.115+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:51.423 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:51.116+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:35:51.689 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:51.659+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:51.689 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:51.659+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:51.690 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:51.659+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:51.690 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:51.660+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:35:51.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:51.688+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:51.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:51.688+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:51.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:51.688+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:51.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:51.689+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:35:52.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:52.145+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:52.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:52.145+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:52.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:52.145+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:52.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:52.146+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:35:52.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:52.730+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:52.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:52.730+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:52.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:52.730+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:52.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:52.731+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:35:52.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:52.654+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:52.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:52.654+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:52.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:52.654+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:52.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:52.655+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:35:53.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:53.124+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:53.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:53.124+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:53.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:53.124+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:53.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:53.125+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:35:53.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:53.613+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:53.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:53.613+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:53.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:53.613+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:53.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:53.613+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:35:53.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:53.782+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:53.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:53.782+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:53.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:53.782+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:53.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:53.782+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:35:54.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:54.078+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:54.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:54.078+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:54.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:54.078+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:54.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:54.078+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:35:54.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:54.616+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:54.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:54.616+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:54.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:54.616+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:54.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:54.617+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:35:54.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:54.737+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:54.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:54.737+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:54.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:54.737+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:54.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:54.738+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:35:55.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:55.121+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:55.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:55.121+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:55.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:55.121+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:55.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:55.122+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:35:55.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:55.752+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:55.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:55.752+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:55.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:55.752+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:55.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:55.753+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:35:55.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:55.584+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:55.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:55.584+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:55.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:55.584+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:55.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:55.585+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:35:56.424 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:35:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:35:56.299+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 16753 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:35:56.424 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:56.171+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:56.424 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:56.171+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:56.425 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:56.171+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:56.425 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:56.171+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:35:56.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:56.582+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:56.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:56.582+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:56.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:56.582+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:56.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:56.583+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:35:57.126 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:56.763+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:57.126 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:56.763+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:57.126 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:56.763+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:57.126 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:56.764+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:35:57.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:57.125+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:57.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:57.125+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:57.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:57.125+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:57.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:57.126+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:35:57.814 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:57.559+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:57.814 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:57.559+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:57.815 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:57.559+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:57.815 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:57.560+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:35:58.080 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:57.814+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:58.081 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:57.814+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:58.081 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:57.814+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:58.081 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:57.815+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:35:58.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:58.080+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:58.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:58.080+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:58.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:58.080+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:58.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:58.081+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:35:58.861 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:58.563+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:58.862 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:58.563+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:58.862 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:58.563+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:58.862 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:58.565+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:35:59.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:58.861+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:59.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:58.861+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:59.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:58.861+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:35:59.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:58.862+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:35:59.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:59.104+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:59.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:59.104+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:59.199 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:59.104+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:35:59.199 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:35:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:35:59.105+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:35:59.894 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:59.595+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:59.895 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:59.595+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:59.895 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:59.595+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:35:59.895 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:35:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:35:59.596+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:36:00.161 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:59.893+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:00.161 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:59.893+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:00.161 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:59.893+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:00.161 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:35:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:35:59.894+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:36:00.162 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:00.127+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:00.162 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:00.127+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:00.162 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:00.127+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:00.162 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:00.128+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:36:00.897 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:00.616+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:00.897 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:00.616+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:00.897 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:00.616+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:00.897 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:00.616+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:36:01.147 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:00.896+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:01.147 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:00.896+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:01.147 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:00.896+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:01.148 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:00.896+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:36:01.427 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:36:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:36:01.300+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 16841 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:36:01.427 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:01.153+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:01.427 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:01.153+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:01.428 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:01.153+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:01.428 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:01.154+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:36:01.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:01.654+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:01.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:01.654+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:01.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:01.654+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:01.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:01.655+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:36:02.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:02.124+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:02.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:02.124+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:02.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:02.124+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:02.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:02.125+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:36:02.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:01.912+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:02.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:01.912+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:02.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:01.912+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:02.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:01.913+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:36:02.882 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:02.612+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:02.883 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:02.612+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:02.883 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:02.612+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:02.883 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:02.613+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:36:03.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:03.096+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:03.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:03.096+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:03.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:03.096+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:03.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:03.097+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:36:03.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:02.881+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:03.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:02.881+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:03.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:02.881+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:03.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:02.882+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:36:03.859 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:03.616+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:03.859 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:03.616+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:03.859 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:03.616+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:03.859 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:03.617+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:36:04.117 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:03.858+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:04.117 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:03.858+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:04.118 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:03.858+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:04.118 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:03.858+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:36:04.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:04.116+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:04.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:04.116+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:04.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:04.116+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:04.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:04.117+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:36:04.856 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:04.604+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:04.856 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:04.604+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:04.857 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:04.604+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:04.857 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:04.605+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:36:05.127 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:04.854+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:05.128 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:04.854+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:05.128 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:04.854+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:05.128 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:04.855+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:36:05.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:05.126+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:05.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:05.126+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:05.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:05.126+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:05.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:05.127+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:36:05.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:05.860+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:05.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:05.860+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:05.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:05.860+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:05.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:05.860+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:36:05.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:05.632+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:05.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:05.632+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:05.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:05.632+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:05.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:05.633+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:36:06.425 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:36:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:36:06.301+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 16931 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:36:06.425 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:06.111+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:06.425 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:06.111+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:06.426 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:06.111+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:06.426 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:06.112+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:36:06.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:06.636+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:06.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:06.636+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:06.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:06.636+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:06.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:06.636+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:36:07.113 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:06.856+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:07.113 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:06.856+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:07.113 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:06.856+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:07.114 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:06.857+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:36:07.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:07.111+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:07.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:07.111+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:07.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:07.111+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:07.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:07.112+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:36:07.884 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:07.611+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:07.885 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:07.611+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:07.885 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:07.611+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:07.885 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:07.611+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:36:08.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:08.117+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:08.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:08.117+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:08.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:08.117+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:08.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:08.118+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:36:08.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:07.883+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:08.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:07.883+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:08.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:07.883+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:08.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:07.884+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:36:08.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:08.605+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:08.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:08.605+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:08.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:08.605+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:08.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:08.606+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:36:08.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:08.837+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:08.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:08.837+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:08.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:08.837+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:08.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:08.838+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:36:09.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:09.151+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:09.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:09.151+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:09.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:09.151+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:09.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:09.152+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:36:09.833 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:09.574+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:09.833 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:09.574+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:09.833 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:09.574+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:09.833 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:09.575+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:36:10.153 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:09.860+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:10.153 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:09.860+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:10.154 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:09.860+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:10.154 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:09.861+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:36:10.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:10.151+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:10.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:10.151+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:10.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:10.151+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:10.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:10.152+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:36:10.886 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:10.612+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:10.886 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:10.612+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:10.886 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:10.612+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:10.886 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:10.613+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:36:11.149 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:10.884+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:11.150 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:10.884+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:11.150 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:10.884+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:11.150 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:10.885+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:36:11.431 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:36:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:36:11.303+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 17019 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:36:11.431 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:11.146+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:11.432 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:11.146+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:11.432 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:11.146+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:11.432 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:11.147+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:36:11.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:11.630+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:11.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:11.630+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:11.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:11.630+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:11.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:11.631+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:36:12.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:12.101+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:12.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:12.101+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:12.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:12.101+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:12.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:12.102+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:36:12.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:11.853+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:12.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:11.853+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:12.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:11.853+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:12.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:11.854+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:36:12.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:12.821+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:12.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:12.821+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:12.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:12.821+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:12.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:12.822+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:36:12.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:12.654+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:12.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:12.654+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:12.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:12.654+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:12.949 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:12.655+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:36:13.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:13.059+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:13.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:13.059+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:13.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:13.059+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:13.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:13.059+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:36:13.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:13.843+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:13.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:13.843+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:13.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:13.843+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:13.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:13.844+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:36:13.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:13.685+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:13.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:13.685+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:13.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:13.685+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:13.949 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:13.686+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:36:14.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:14.084+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:14.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:14.084+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:14.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:14.084+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:14.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:14.085+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:36:15.041 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:14.731+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:15.041 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:14.731+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:15.042 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:14.731+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:15.042 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:14.731+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:36:15.042 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:14.839+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:15.042 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:14.839+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:15.043 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:14.840+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:15.043 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:14.840+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:36:15.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:15.039+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:15.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:15.039+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:15.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:15.039+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:15.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:15.040+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:36:16.090 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:15.764+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:16.090 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:15.764+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:16.091 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:15.764+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:16.091 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:15.765+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:36:16.091 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:15.869+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:16.091 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:15.869+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:16.091 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:15.869+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:16.091 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:15.870+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:36:16.431 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:36:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:36:16.305+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 17110 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:36:16.431 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:16.089+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:16.431 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:16.089+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:16.431 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:16.089+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:16.431 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:16.090+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:36:17.097 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:16.716+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:17.097 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:16.716+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:17.097 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:16.716+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:17.097 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:16.717+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:36:17.097 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:16.830+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:17.097 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:16.830+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:17.098 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:16.830+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:17.098 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:16.831+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:36:17.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:17.095+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:17.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:17.095+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:17.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:17.095+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:17.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:17.096+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:36:18.080 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:17.738+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:18.081 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:17.738+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:18.081 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:17.738+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:18.081 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:17.739+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:36:18.081 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:17.804+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:18.081 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:17.804+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:18.081 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:17.804+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:18.082 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:17.805+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:36:18.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:18.079+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:18.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:18.079+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:18.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:18.079+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:18.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:18.080+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:36:19.082 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:18.742+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:19.083 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:18.742+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:19.083 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:18.742+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:19.083 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:18.743+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:36:19.083 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:18.850+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:19.083 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:18.850+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:19.083 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:18.850+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:19.084 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:18.851+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:36:19.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:19.081+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:19.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:19.081+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:19.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:19.081+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:19.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:19.082+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:36:20.053 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:19.790+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:20.054 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:19.790+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:20.054 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:19.790+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:20.054 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:19.791+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:36:20.055 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:19.874+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:20.055 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:19.874+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:20.056 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:19.874+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:20.056 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:19.875+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:36:20.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:20.051+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:20.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:20.051+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:20.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:20.051+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:20.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:20.052+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:36:21.060 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:20.780+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:21.060 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:20.780+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:21.061 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:20.780+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:21.061 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:20.781+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:36:21.061 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:20.866+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:21.061 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:20.866+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:21.061 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:20.866+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:21.061 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:20.867+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:36:21.322 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:36:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:36:21.307+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 17198 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:36:21.323 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:21.059+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:21.323 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:21.059+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:21.323 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:21.059+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:21.323 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:21.059+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:36:22.099 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:21.823+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:22.099 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:21.823+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:22.099 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:21.823+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:22.099 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:21.824+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:36:22.099 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:21.914+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:22.100 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:21.914+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:22.100 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:21.914+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:22.100 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:21.915+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:36:22.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:22.097+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:22.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:22.097+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:22.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:22.097+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:22.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:22.098+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:36:22.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:22.874+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:22.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:22.874+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:22.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:22.874+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:22.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:22.875+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:36:22.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:22.833+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:22.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:22.833+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:22.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:22.833+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:22.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:22.833+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:36:23.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:23.063+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:23.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:23.063+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:23.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:23.063+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:23.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:23.064+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:36:24.093 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:23.838+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:24.093 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:23.838+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:24.093 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:23.838+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:24.093 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:23.839+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:36:24.093 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:23.869+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:24.094 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:23.869+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:24.094 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:23.869+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:24.094 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:23.870+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:36:24.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:24.091+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:24.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:24.091+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:24.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:24.091+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:24.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:24.092+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:36:25.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:25.054+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:25.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:25.054+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:25.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:25.054+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:25.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:25.054+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:36:25.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:24.879+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:25.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:24.879+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:25.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:24.879+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:25.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:24.880+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:36:25.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:24.894+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:25.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:24.894+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:25.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:24.894+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:25.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:24.895+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:36:26.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:25.893+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:26.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:25.893+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:26.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:25.893+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:26.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:25.894+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:36:26.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:26.008+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:26.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:26.008+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:26.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:26.008+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:26.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:26.009+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:36:26.199 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:25.893+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:26.199 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:25.893+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:26.199 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:25.893+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:26.200 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:25.893+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:36:26.697 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:36:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:36:26.309+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 17288 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:36:27.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:26.915+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:27.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:26.915+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:27.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:26.915+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:27.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:26.915+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:36:27.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:27.026+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:27.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:27.026+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:27.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:27.026+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:27.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:27.026+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:36:27.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:26.943+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:27.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:26.943+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:27.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:26.943+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:27.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:26.943+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:36:28.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:27.928+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:28.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:27.928+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:28.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:27.928+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:28.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:27.929+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:36:28.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:28.028+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:28.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:28.028+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:28.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:28.028+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:28.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:28.029+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:36:28.199 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:27.950+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:28.199 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:27.950+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:28.199 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:27.950+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:28.199 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:27.951+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:36:29.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:29.073+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:29.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:29.073+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:29.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:29.073+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:29.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:29.073+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:36:29.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:28.948+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:29.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:28.948+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:29.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:28.948+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:29.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:28.949+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:36:29.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:28.950+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:29.449 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:28.950+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:29.449 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:28.950+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:29.449 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:28.950+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:36:30.162 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:30.025+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:30.162 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:30.025+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:30.162 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:30.025+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:30.162 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:30.026+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:36:30.163 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:29.953+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:30.163 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:29.953+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:30.163 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:29.953+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:30.163 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:29.953+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:36:30.163 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:29.967+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:30.163 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:29.967+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:30.164 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:29.967+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:30.164 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:29.968+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:36:31.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:31.037+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:31.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:31.037+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:31.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:31.037+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:31.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:31.038+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:36:31.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:30.987+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:31.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:30.987+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:31.199 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:30.987+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:31.199 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:30.987+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:36:31.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:30.954+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:31.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:30.954+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:31.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:30.954+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:31.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:30.955+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:36:31.697 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:36:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:36:31.311+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 17376 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:36:32.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:31.938+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:32.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:31.938+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:32.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:31.938+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:32.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:31.939+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:36:32.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:32.000+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:32.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:32.000+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:32.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:32.000+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:32.199 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:32.001+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:36:32.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:31.943+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:32.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:31.943+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:32.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:31.943+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:32.200 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:31.944+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:36:33.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:32.913+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:33.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:32.913+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:33.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:32.913+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:33.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:32.913+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:36:33.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:32.943+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:33.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:32.943+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:33.199 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:32.943+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:33.199 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:32.944+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:36:33.199 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:33.019+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:33.199 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:33.019+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:33.199 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:33.019+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:33.199 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:33.020+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:36:34.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:33.933+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:34.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:33.933+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:34.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:33.933+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:34.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:33.933+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:36:34.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:33.994+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:34.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:33.994+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:34.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:33.994+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:34.199 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:33.995+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:36:34.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:33.865+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:34.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:33.865+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:34.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:33.865+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:34.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:33.866+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:36:35.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:34.877+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:35.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:34.877+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:35.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:34.877+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:35.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:34.878+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:36:35.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:34.959+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:35.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:34.959+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:35.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:34.959+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:35.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:34.960+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:36:35.199 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:34.951+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:35.199 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:34.951+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:35.199 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:34.951+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:35.199 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:34.952+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:36:36.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:35.919+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:36.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:35.919+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:36.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:35.919+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:36.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:35.920+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:36:36.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:35.985+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:36.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:35.985+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:36.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:35.985+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:36.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:35.986+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:36:36.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:35.842+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:36.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:35.842+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:36.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:35.842+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:36.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:35.843+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:36:36.697 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:36:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:36:36.313+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 17467 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:36:37.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:37.030+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:37.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:37.030+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:37.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:37.030+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:37.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:37.031+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:36:37.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:36.822+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:37.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:36.822+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:37.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:36.822+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:37.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:36.823+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:36:37.199 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:36.916+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:37.199 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:36.916+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:37.199 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:36.916+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:37.199 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:36.916+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:36:38.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:37.984+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:38.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:37.984+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:38.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:37.984+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:38.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:37.985+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:36:38.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:37.855+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:38.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:37.855+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:38.202 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:37.855+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:38.202 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:37.856+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:36:38.202 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:37.934+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:38.202 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:37.934+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:38.203 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:37.934+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:38.203 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:37.934+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:36:39.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:38.938+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:39.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:38.938+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:39.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:38.938+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:39.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:38.939+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:36:39.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:38.931+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:39.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:38.931+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:39.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:38.931+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:39.199 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:38.932+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:36:39.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:38.836+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:39.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:38.836+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:39.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:38.836+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:39.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:38.837+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:36:40.162 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:39.918+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:40.163 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:39.918+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:40.163 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:39.918+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:40.163 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:39.918+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:36:40.163 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:39.911+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:40.163 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:39.911+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:40.164 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:39.911+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:40.164 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:39.911+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:36:40.164 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:39.836+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:40.164 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:39.836+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:40.164 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:39.836+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:40.164 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:39.837+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:36:41.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:40.915+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:41.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:40.915+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:41.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:40.915+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:41.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:40.915+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:36:41.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:40.874+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:41.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:40.874+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:41.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:40.874+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:41.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:40.875+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:36:41.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:40.857+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:41.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:40.857+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:41.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:40.857+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:41.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:40.858+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:36:41.697 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:36:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:36:41.315+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 17555 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:36:42.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:41.912+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:42.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:41.912+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:42.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:41.912+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:42.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:41.913+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:36:42.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:41.840+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:42.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:41.840+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:42.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:41.840+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:42.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:41.841+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:36:42.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:41.882+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:42.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:41.882+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:42.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:41.882+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:42.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:41.883+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:36:43.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:42.932+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:43.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:42.932+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:43.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:42.932+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:43.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:42.933+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:36:43.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:42.878+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:43.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:42.878+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:43.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:42.878+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:43.199 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:42.879+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:36:43.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:42.928+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:43.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:42.928+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:43.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:42.928+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:43.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:42.929+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:36:44.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:43.969+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:44.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:43.969+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:44.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:43.969+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:44.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:43.970+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:36:44.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:43.847+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:44.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:43.847+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:44.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:43.847+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:44.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:43.848+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:36:44.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:43.900+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:44.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:43.900+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:44.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:43.900+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:44.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:43.901+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:36:45.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:44.946+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:45.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:44.946+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:45.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:44.946+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:45.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:44.947+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:36:45.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:44.886+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:45.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:44.886+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:45.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:44.886+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:45.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:44.887+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:36:45.199 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:44.810+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:45.199 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:44.810+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:45.199 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:44.810+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:45.199 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:44.811+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:36:46.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:45.777+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:46.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:45.777+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:46.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:45.777+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:46.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:45.778+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:36:46.199 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:45.982+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:46.199 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:45.982+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:46.199 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:45.982+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:46.199 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:45.982+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:36:46.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:45.887+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:46.200 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:45.887+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:46.200 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:45.887+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:46.200 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:45.887+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:36:46.697 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:36:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:36:46.316+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 17645 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:36:47.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:46.981+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:47.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:46.981+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:47.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:46.981+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:47.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:46.982+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:36:47.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:46.910+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:47.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:46.910+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:47.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:46.910+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:47.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:46.910+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:36:47.199 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:46.784+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:47.199 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:46.784+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:47.199 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:46.784+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:47.199 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:46.784+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:36:48.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:47.973+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:48.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:47.973+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:48.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:47.973+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:48.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:47.974+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:36:48.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:47.957+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:48.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:47.957+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:48.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:47.957+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:48.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:47.957+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:36:48.199 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:47.772+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:48.199 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:47.772+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:48.199 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:47.772+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:48.199 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:47.773+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:36:49.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:48.991+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:49.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:48.991+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:49.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:48.991+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:49.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:48.992+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:36:49.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:48.806+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:49.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:48.806+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:49.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:48.806+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:49.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:48.807+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:36:49.199 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:48.975+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:49.199 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:48.975+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:49.199 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:48.975+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:49.199 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:48.977+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:36:50.163 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:49.930+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:50.163 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:49.930+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:50.164 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:49.930+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:50.164 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:49.931+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:36:50.164 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:49.769+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:50.164 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:49.769+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:50.164 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:49.769+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:50.165 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:49.770+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:36:50.165 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:50.000+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:50.165 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:50.000+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:50.165 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:50.000+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:50.165 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:50.002+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:36:51.004 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:50.732+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:51.004 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:50.732+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:51.005 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:50.732+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:51.005 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:50.732+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:36:51.005 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:50.964+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:51.005 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:50.964+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:51.005 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:50.964+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:51.005 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:50.965+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:36:51.320 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:51.003+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:51.320 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:51.003+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:51.320 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:51.003+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:51.320 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:51.003+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:36:51.697 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:36:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:36:51.318+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 17733 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:36:52.024 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:51.755+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:52.024 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:51.755+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:52.024 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:51.755+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:52.024 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:51.756+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:36:52.024 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:51.980+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:52.024 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:51.980+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:52.025 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:51.980+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:52.025 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:51.981+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:36:52.435 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:52.022+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:52.435 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:52.022+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:52.435 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:52.022+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:52.435 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:52.023+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:36:52.996 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:52.732+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:52.996 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:52.732+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:52.996 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:52.732+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:52.996 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:52.733+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:36:53.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:53.021+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:53.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:53.021+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:53.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:53.021+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:53.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:53.022+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:36:53.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:52.994+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:53.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:52.994+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:53.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:52.994+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:53.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:52.996+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:36:54.014 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:53.734+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:54.014 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:53.734+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:54.014 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:53.734+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:54.014 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:53.734+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:36:54.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:54.012+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:54.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:54.012+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:54.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:54.012+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:54.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:54.013+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:36:54.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:54.041+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:54.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:54.041+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:54.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:54.041+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:54.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:54.042+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:36:55.031 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:54.994+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:55.032 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:54.994+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:55.032 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:54.994+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:55.032 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:54.995+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:36:55.032 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:54.767+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:55.032 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:54.767+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:55.033 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:54.767+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:55.033 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:54.768+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:36:55.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:55.030+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:55.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:55.030+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:55.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:55.030+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:55.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:55.031+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:36:56.011 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:55.741+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:56.012 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:55.741+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:56.012 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:55.741+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:56.012 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:55.742+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:36:56.322 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:56.043+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:56.322 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:56.043+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:56.322 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:56.043+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:56.322 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:56.044+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:36:56.322 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:56.010+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:56.323 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:56.010+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:56.323 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:56.010+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:56.323 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:56.011+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:36:56.697 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:36:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:36:56.320+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 17824 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:36:57.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:57.031+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:57.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:57.031+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:57.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:57.031+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:57.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:57.032+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:36:57.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:56.791+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:57.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:56.791+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:57.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:56.791+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:57.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:56.792+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:36:57.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:56.977+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:57.199 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:56.977+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:57.199 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:56.977+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:57.199 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:56.977+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:36:58.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:57.959+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:58.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:57.959+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:58.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:57.959+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:58.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:57.959+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:36:58.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:57.822+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:58.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:57.822+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:58.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:57.822+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:58.199 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:57.823+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:36:58.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:57.984+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:58.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:57.984+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:58.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:57.984+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:58.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:57.985+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:36:59.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:58.984+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:59.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:58.984+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:59.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:58.984+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:36:59.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:58.985+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:36:59.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:58.995+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:59.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:58.995+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:59.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:58.995+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:36:59.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:36:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:36:58.995+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:36:59.199 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:58.851+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:59.199 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:58.851+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:59.199 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:58.851+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:36:59.199 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:58.852+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:37:00.164 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:59.992+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:00.165 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:59.992+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:00.165 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:59.992+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:00.165 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:36:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:36:59.992+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:37:00.166 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:59.857+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:00.166 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:59.857+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:00.166 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:59.857+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:00.166 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:36:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:36:59.857+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:37:00.166 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:00.044+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:00.167 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:00.044+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:00.167 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:00.044+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:00.167 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:00.045+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:37:01.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:00.988+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:01.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:00.988+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:01.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:00.988+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:01.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:00.988+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:37:01.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:01.076+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:01.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:01.076+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:01.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:01.076+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:01.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:01.076+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:37:01.199 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:00.886+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:01.200 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:00.886+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:01.200 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:00.886+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:01.200 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:00.887+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:37:01.697 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:37:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:37:01.322+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 17912 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:37:02.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:01.901+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:02.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:01.901+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:02.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:01.901+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:02.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:01.901+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:37:02.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:01.941+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:02.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:01.941+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:02.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:01.941+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:02.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:01.942+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:37:02.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:02.045+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:02.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:02.045+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:02.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:02.045+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:02.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:02.046+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:37:02.955 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:02.900+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:02.956 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:02.900+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:02.956 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:02.900+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:02.956 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:02.900+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:37:03.351 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:02.954+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:03.352 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:02.954+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:03.352 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:02.954+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:03.352 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:02.955+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:37:03.353 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:03.055+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:03.353 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:03.055+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:03.353 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:03.055+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:03.353 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:03.056+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:37:04.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:03.874+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:04.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:03.875+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:04.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:03.875+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:04.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:03.875+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:37:04.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:03.935+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:04.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:03.935+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:04.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:03.935+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:04.199 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:03.936+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:37:04.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:04.071+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:04.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:04.071+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:04.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:04.071+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:04.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:04.072+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:37:05.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:04.942+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:05.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:04.942+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:05.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:04.942+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:05.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:04.942+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:37:05.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:04.879+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:05.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:04.879+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:05.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:04.879+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:05.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:04.879+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:37:05.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:05.074+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:05.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:05.074+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:05.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:05.074+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:05.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:05.075+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:37:06.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:06.053+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:06.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:06.053+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:06.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:06.053+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:06.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:06.054+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:37:06.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:05.908+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:06.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:05.908+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:06.199 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:05.908+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:06.199 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:05.909+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:37:06.199 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:05.897+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:06.199 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:05.897+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:06.199 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:05.897+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:06.199 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:05.897+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:37:06.697 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:37:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:37:06.324+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 18002 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:37:07.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:06.870+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:07.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:06.870+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:07.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:06.870+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:07.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:06.871+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:37:07.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:07.076+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:07.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:07.076+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:07.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:07.076+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:07.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:07.077+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:37:07.199 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:06.870+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:07.199 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:06.870+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:07.199 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:06.870+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:07.199 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:06.871+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:37:08.113 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:07.852+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:08.114 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:07.852+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:08.114 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:07.852+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:08.114 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:07.852+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:37:08.114 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:07.833+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:08.114 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:07.833+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:08.114 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:07.833+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:08.115 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:07.833+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:37:08.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:08.112+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:08.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:08.112+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:08.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:08.112+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:08.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:08.112+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:37:09.110 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:08.808+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:09.111 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:08.808+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:09.111 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:08.808+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:09.111 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:08.809+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:37:09.111 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:08.840+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:09.111 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:08.840+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:09.112 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:08.841+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:09.112 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:08.841+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:37:09.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:09.109+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:09.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:09.109+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:09.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:09.109+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:09.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:09.110+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:37:10.087 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:09.879+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:10.087 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:09.879+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:10.087 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:09.879+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:10.087 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:09.880+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:37:10.088 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:09.849+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:10.088 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:09.849+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:10.088 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:09.849+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:10.088 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:09.849+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:37:10.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:10.085+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:10.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:10.085+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:10.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:10.085+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:10.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:10.086+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:37:11.126 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:10.843+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:11.126 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:10.843+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:11.127 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:10.843+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:11.127 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:10.844+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:37:11.127 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:10.848+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:11.127 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:10.848+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:11.127 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:10.848+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:11.127 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:10.849+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:37:11.447 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:37:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:37:11.326+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 18090 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:37:11.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:11.125+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:11.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:11.125+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:11.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:11.125+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:11.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:11.125+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:37:12.149 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:11.847+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:12.149 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:11.847+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:12.149 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:11.847+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:12.150 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:11.848+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:37:12.150 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:11.871+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:12.150 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:11.871+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:12.150 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:11.871+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:12.150 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:11.872+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:37:12.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:12.147+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:12.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:12.147+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:12.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:12.147+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:12.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:12.148+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:37:13.109 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:12.908+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:13.109 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:12.908+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:13.109 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:12.908+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:13.110 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:12.909+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:37:13.110 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:12.799+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:13.110 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:12.799+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:13.110 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:12.799+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:13.110 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:12.800+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:37:13.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:13.108+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:13.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:13.108+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:13.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:13.108+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:13.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:13.109+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:37:14.143 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:13.771+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:14.143 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:13.771+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:14.143 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:13.771+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:14.143 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:13.772+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:37:14.143 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:13.949+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:14.144 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:13.949+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:14.144 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:13.949+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:14.144 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:13.950+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:37:14.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:14.141+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:14.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:14.141+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:14.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:14.141+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:14.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:14.142+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:37:15.115 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:14.814+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:15.115 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:14.814+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:15.115 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:14.814+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:15.116 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:14.815+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:37:15.116 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:14.911+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:15.116 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:14.911+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:15.116 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:14.911+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:15.116 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:14.912+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:37:15.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:15.113+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:15.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:15.113+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:15.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:15.113+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:15.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:15.114+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:37:16.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:15.879+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:16.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:15.879+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:16.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:15.879+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:16.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:15.880+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:37:16.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:16.066+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:16.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:16.066+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:16.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:16.066+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:16.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:16.067+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:37:16.199 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:15.841+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:16.199 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:15.841+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:16.199 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:15.841+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:16.199 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:15.841+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:37:16.697 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:37:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:37:16.328+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 18181 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:37:17.110 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:16.848+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:17.110 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:16.848+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:17.110 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:16.848+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:17.110 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:16.849+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:37:17.111 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:16.862+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:17.111 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:16.862+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:17.111 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:16.862+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:17.111 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:16.863+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:37:17.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:17.108+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:17.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:17.108+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:17.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:17.108+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:17.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:17.109+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:37:18.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:17.851+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:18.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:17.851+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:18.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:17.851+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:18.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:17.852+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:37:18.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:18.094+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:18.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:18.094+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:18.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:18.094+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:18.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:18.095+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:37:18.199 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:17.859+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:18.199 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:17.859+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:18.199 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:17.859+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:18.199 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:17.860+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:37:19.072 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:18.818+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:19.073 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:18.818+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:19.073 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:18.818+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:19.074 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:18.819+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:37:19.074 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:18.812+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:19.074 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:18.812+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:19.075 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:18.812+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:19.075 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:18.812+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:37:19.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:19.070+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:19.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:19.070+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:19.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:19.070+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:19.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:19.071+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:37:20.123 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:19.812+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:20.123 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:19.812+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:20.123 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:19.812+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:20.123 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:19.813+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:37:20.124 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:19.775+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:20.124 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:19.775+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:20.124 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:19.775+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:20.124 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:19.776+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:37:20.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:20.121+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:20.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:20.121+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:20.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:20.121+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:20.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:20.122+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:37:21.084 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:20.786+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:21.084 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:20.786+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:21.084 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:20.786+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:21.084 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:20.787+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:37:21.085 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:20.825+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:21.085 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:20.825+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:21.085 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:20.825+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:21.085 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:20.826+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:37:21.345 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:37:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:37:21.330+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 18268 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:37:21.346 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:21.082+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:21.346 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:21.082+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:21.346 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:21.082+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:21.346 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:21.083+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:37:22.108 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:21.826+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:22.109 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:21.826+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:22.109 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:21.826+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:22.109 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:21.827+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:37:22.109 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:21.838+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:22.109 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:21.838+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:22.110 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:21.838+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:22.110 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:21.839+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:37:22.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:22.107+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:22.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:22.107+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:22.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:22.107+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:22.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:22.108+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:37:23.106 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:22.801+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:23.106 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:22.801+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:23.106 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:22.801+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:23.106 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:22.802+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:37:23.107 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:22.869+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:23.107 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:22.869+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:23.107 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:22.869+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:23.107 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:22.870+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:37:23.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:23.104+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:23.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:23.104+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:23.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:23.104+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:23.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:23.105+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:37:24.093 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:23.824+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:24.094 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:23.824+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:24.094 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:23.824+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:24.094 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:23.824+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:37:24.094 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:23.845+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:24.094 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:23.845+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:24.095 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:23.845+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:24.095 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:23.845+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:37:24.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:24.092+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:24.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:24.092+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:24.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:24.092+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:24.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:24.093+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:37:25.081 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:24.801+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:25.081 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:24.801+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:25.081 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:24.801+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:25.081 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:24.802+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:37:25.081 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:24.835+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:25.082 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:24.835+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:25.082 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:24.835+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:25.082 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:24.835+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:37:25.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:25.079+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:25.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:25.079+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:25.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:25.079+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:25.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:25.080+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:37:26.091 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:25.758+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:26.091 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:25.758+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:26.091 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:25.758+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:26.091 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:25.759+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:37:26.091 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:25.793+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:26.092 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:25.793+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:26.092 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:25.793+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:26.092 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:25.793+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:37:26.349 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:26.089+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:26.349 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:26.089+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:26.349 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:26.089+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:26.349 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:26.090+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:37:26.350 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:37:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:37:26.332+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 18357 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:37:27.083 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:26.780+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:27.084 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:26.780+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:27.084 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:26.780+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:27.084 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:26.780+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:37:27.084 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:26.779+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:27.084 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:26.779+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:27.084 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:26.779+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:27.085 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:26.780+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:37:27.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:27.082+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:27.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:27.082+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:27.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:27.082+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:27.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:27.084+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:37:28.095 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:27.770+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:28.095 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:27.770+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:28.095 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:27.770+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:28.096 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:27.771+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:37:28.096 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:27.810+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:28.096 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:27.810+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:28.096 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:27.810+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:28.096 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:27.811+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:37:28.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:28.094+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:28.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:28.094+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:28.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:28.094+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:28.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:28.095+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:37:29.094 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:28.816+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:29.094 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:28.816+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:29.094 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:28.816+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:29.094 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:28.816+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:37:29.094 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:28.807+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:29.095 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:28.807+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:29.095 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:28.807+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:29.095 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:28.808+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:37:29.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:29.093+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:29.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:29.093+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:29.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:29.093+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:29.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:29.094+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:37:30.062 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:29.811+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:30.062 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:29.811+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:30.062 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:29.811+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:30.062 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:29.811+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:37:30.063 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:29.852+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:30.063 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:29.852+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:30.063 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:29.852+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:30.063 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:29.853+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:37:30.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:30.061+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:30.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:30.061+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:30.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:30.061+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:30.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:30.062+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:37:31.094 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:30.833+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:31.095 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:30.833+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:31.095 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:30.833+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:31.095 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:30.833+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:37:31.095 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:30.808+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:31.095 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:30.808+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:31.095 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:30.808+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:31.096 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:30.808+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:37:31.348 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:37:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:37:31.335+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 18445 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:37:31.348 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:31.092+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:31.348 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:31.093+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:31.348 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:31.093+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:31.349 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:31.094+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:37:32.144 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:31.838+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:32.144 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:31.838+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:32.144 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:31.838+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:32.145 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:31.838+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:37:32.145 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:31.857+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:32.145 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:31.857+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:32.145 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:31.857+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:32.145 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:31.858+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:37:32.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:32.142+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:32.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:32.142+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:32.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:32.142+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:32.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:32.144+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:37:33.165 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:32.859+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:33.165 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:32.859+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:33.166 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:32.859+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:33.166 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:32.860+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:37:33.166 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:32.845+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:33.166 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:32.845+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:33.166 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:32.845+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:33.166 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:32.846+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:37:33.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:33.164+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:33.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:33.164+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:33.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:33.164+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:33.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:33.164+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:37:34.172 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:33.889+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:34.173 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:33.889+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:34.173 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:33.889+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:34.173 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:33.890+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:37:34.173 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:33.821+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:34.173 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:33.821+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:34.174 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:33.821+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:34.174 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:33.821+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:37:34.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:34.171+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:34.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:34.171+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:34.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:34.171+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:34.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:34.172+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:37:35.130 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:34.907+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:35.130 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:34.907+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:35.130 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:34.907+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:35.130 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:34.908+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:37:35.131 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:34.811+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:35.131 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:34.811+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:35.131 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:34.811+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:35.131 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:34.812+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:37:35.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:35.128+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:35.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:35.128+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:35.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:35.128+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:35.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:35.129+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:37:36.149 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:35.853+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:36.150 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:35.853+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:36.150 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:35.853+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:36.150 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:35.854+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:37:36.150 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:35.930+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:36.150 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:35.930+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:36.151 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:35.930+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:36.151 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:35.931+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:37:36.421 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:37:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:37:36.336+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 18536 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:37:36.421 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:36.148+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:36.422 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:36.148+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:36.422 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:36.148+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:36.422 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:36.149+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:37:37.188 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:36.851+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:37.188 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:36.851+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:37.188 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:36.851+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:37.188 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:36.852+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:37:37.189 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:36.975+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:37.189 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:36.975+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:37.189 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:36.975+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:37.189 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:36.976+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:37:37.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:37.186+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:37.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:37.186+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:37.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:37.186+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:37.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:37.187+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:37:38.159 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:38.011+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:38.159 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:38.011+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:38.159 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:38.011+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:38.159 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:38.012+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:37:38.159 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:37.876+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:38.159 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:37.877+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:38.160 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:37.877+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:38.160 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:37.877+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:37:38.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:38.157+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:38.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:38.157+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:38.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:38.157+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:38.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:38.158+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:37:39.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:38.966+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:39.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:38.966+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:39.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:38.966+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:39.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:38.967+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:37:39.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:38.829+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:39.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:38.829+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:39.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:38.829+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:39.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:38.829+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:37:39.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:39.195+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:39.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:39.195+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:39.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:39.195+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:39.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:39.196+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:37:40.166 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:39.789+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:40.166 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:39.789+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:40.166 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:39.789+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:40.166 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:39.790+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:37:40.166 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:39.973+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:40.167 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:39.973+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:40.167 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:39.973+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:40.167 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:39.974+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:37:40.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:40.185+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:40.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:40.185+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:40.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:40.185+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:40.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:40.186+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:37:41.022 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:40.755+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:41.022 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:40.755+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:41.022 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:40.755+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:41.022 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:40.756+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:37:41.340 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:41.227+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:41.341 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:41.227+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:41.341 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:41.227+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:41.341 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:41.228+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:37:41.341 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:41.020+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:41.341 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:41.020+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:41.342 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:41.020+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:41.342 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:41.021+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:37:41.697 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:37:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:37:41.339+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 18624 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:37:42.041 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:41.769+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:42.041 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:41.769+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:42.041 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:41.769+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:42.041 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:41.770+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:37:42.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:42.039+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:42.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:42.039+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:42.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:42.039+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:42.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:42.040+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:37:42.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:42.204+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:42.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:42.204+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:42.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:42.204+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:42.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:42.204+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:37:43.031 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:42.748+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:43.031 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:42.748+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:43.031 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:42.748+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:43.031 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:42.749+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:37:43.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:43.029+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:43.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:43.029+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:43.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:43.029+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:43.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:43.030+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:37:43.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:43.243+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:43.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:43.243+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:43.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:43.243+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:43.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:43.244+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:37:44.056 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:43.732+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:44.057 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:43.732+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:44.057 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:43.732+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:44.057 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:43.733+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:37:44.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:44.055+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:44.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:44.055+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:44.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:44.055+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:44.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:44.056+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:37:44.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:44.276+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:44.449 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:44.276+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:44.449 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:44.276+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:44.449 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:44.276+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:37:45.055 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:44.721+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:45.056 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:44.721+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:45.056 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:44.721+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:45.056 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:44.722+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:37:45.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:45.054+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:45.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:45.054+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:45.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:45.054+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:45.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:45.055+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:37:45.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:45.228+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:45.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:45.228+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:45.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:45.228+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:45.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:45.229+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:37:46.098 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:45.749+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:46.098 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:45.749+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:46.098 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:45.749+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:46.098 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:45.749+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:37:46.354 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:46.096+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:46.354 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:46.096+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:46.354 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:46.096+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:46.354 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:46.097+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:37:46.355 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:46.201+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:46.355 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:46.201+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:46.355 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:46.201+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:46.355 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:46.202+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:37:46.355 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:37:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:37:46.341+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 18714 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:37:47.131 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:46.723+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:47.131 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:46.723+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:47.132 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:46.723+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:47.132 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:46.724+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:37:47.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:47.130+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:47.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:47.130+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:47.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:47.130+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:47.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:47.131+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:37:47.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:47.208+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:47.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:47.208+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:47.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:47.208+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:47.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:47.208+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:37:48.176 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:47.711+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:48.176 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:47.711+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:48.176 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:47.711+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:48.176 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:47.712+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:37:48.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:48.186+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:48.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:48.186+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:48.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:48.186+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:48.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:48.188+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:37:48.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:48.174+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:48.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:48.174+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:48.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:48.174+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:48.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:48.175+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:37:49.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:48.718+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:49.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:48.718+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:49.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:48.718+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:49.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:48.719+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:37:49.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:49.207+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:49.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:49.207+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:49.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:49.207+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:49.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:49.208+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:37:49.699 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:49.199+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:49.699 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:49.199+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:49.700 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:49.199+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:49.700 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:49.200+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:37:50.160 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:49.760+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:50.160 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:49.760+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:50.160 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:49.760+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:50.160 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:49.761+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:37:50.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:50.158+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:50.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:50.158+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:50.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:50.158+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:50.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:50.159+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:37:50.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:50.209+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:50.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:50.209+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:50.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:50.209+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:50.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:50.210+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:37:51.178 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:50.788+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:51.178 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:50.788+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:51.178 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:50.788+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:51.178 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:50.789+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:37:51.447 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:37:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:37:51.343+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 18802 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:37:51.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:51.176+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:51.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:51.176+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:51.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:51.176+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:51.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:51.177+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:37:51.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:51.233+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:51.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:51.233+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:51.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:51.233+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:51.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:51.234+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:37:52.137 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:51.756+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:52.138 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:51.756+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:52.138 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:51.756+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:52.138 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:51.757+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:37:52.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:52.252+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:52.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:52.252+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:52.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:52.252+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:52.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:52.253+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:37:52.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:52.136+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:52.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:52.136+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:52.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:52.136+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:52.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:52.137+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:37:53.144 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:52.745+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:53.144 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:52.745+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:53.144 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:52.745+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:53.144 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:52.746+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:37:53.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:53.215+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:53.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:53.215+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:53.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:53.215+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:53.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:53.215+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:37:53.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:53.142+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:53.449 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:53.142+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:53.449 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:53.142+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:53.449 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:53.144+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:37:54.170 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:53.708+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:54.170 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:53.708+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:54.170 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:53.708+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:54.170 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:53.709+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:37:54.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:54.177+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:54.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:54.177+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:54.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:54.177+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:54.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:54.179+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:37:54.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:54.169+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:54.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:54.169+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:54.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:54.169+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:54.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:54.169+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:37:54.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:54.684+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:54.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:54.684+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:54.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:54.684+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:54.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:54.685+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:37:55.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:55.160+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:55.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:55.160+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:55.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:55.160+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:21.874326+0000 (oldest deadline 2024-05-04T12:27:41.874326+0000) 2024-05-04T12:37:55.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:55.161+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:37:55.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:55.131+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:55.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:55.131+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:55.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:55.131+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:55.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:55.132+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:37:55.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:55.664+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:55.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:55.664+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:55.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:55.664+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:55.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:55.664+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:37:56.447 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:37:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:37:56.345+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 18892 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:37:56.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:56.142+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:37:56.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:56.166+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:56.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:56.166+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:56.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:56.166+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:56.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:56.166+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:37:56.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:56.623+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:56.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:56.623+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:56.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:56.623+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:56.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:56.623+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:37:57.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:57.141+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:37:57.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:57.189+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:57.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:57.189+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:57.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:57.189+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:57.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:57.190+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:37:57.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:57.635+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:57.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:57.635+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:57.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:57.635+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:57.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:57.636+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:37:58.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:58.185+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:37:58.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:58.194+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:58.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:58.194+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:58.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:58.194+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:58.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:58.195+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:37:58.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:58.678+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:58.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:58.678+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:58.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:58.678+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:58.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:58.679+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:37:59.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:37:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:37:59.164+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:37:59.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:59.219+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:59.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:59.219+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:59.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:59.219+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:37:59.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:37:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:37:59.220+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:37:59.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:59.630+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:59.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:59.630+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:59.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:59.630+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:37:59.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:37:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:37:59.630+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:38:00.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:00.200+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:38:00.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:00.202+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:38:00.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:00.202+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:38:00.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:00.202+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:38:00.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:00.203+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:38:00.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:00.581+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:38:00.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:00.581+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:38:00.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:00.581+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:38:00.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:00.582+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:38:01.447 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:38:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:38:01.347+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 18980 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:38:01.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:01.247+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:38:01.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:01.195+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:38:01.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:01.195+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:38:01.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:01.195+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:25.765799+0000 (oldest deadline 2024-05-04T12:27:45.765799+0000) 2024-05-04T12:38:01.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:01.196+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:38:01.950 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:01.540+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:38:01.950 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:01.540+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:38:01.950 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:01.540+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:38:01.950 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:01.541+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:38:02.565 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:02.260+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:38:02.565 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:02.222+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:38:02.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:02.564+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:38:02.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:02.564+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:38:02.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:02.564+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:38:02.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:02.564+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:38:03.615 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:03.262+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:38:03.616 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:03.241+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:38:03.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:03.614+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:38:03.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:03.614+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:38:03.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:03.614+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:38:03.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:03.615+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:38:04.605 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:04.199+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:38:04.606 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:04.264+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:38:04.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:04.604+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:38:04.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:04.604+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:38:04.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:04.604+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:38:04.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:04.604+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:38:05.560 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:05.202+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:38:05.560 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:05.312+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:38:05.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:05.559+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:38:05.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:05.559+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:38:05.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:05.559+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:38:05.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:05.560+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:38:06.447 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:38:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:38:06.349+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 19070 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:38:06.449 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:06.176+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:38:06.450 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:06.355+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:38:06.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:06.590+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:38:06.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:06.590+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:38:06.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:06.590+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:27:34.981665+0000 (oldest deadline 2024-05-04T12:27:54.981665+0000) 2024-05-04T12:38:06.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:06.590+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:38:07.602 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:07.224+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:38:07.602 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:07.394+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:38:07.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:07.600+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:38:08.643 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:08.242+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:38:08.643 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:08.372+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:38:08.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:08.641+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:38:09.675 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:09.239+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:38:09.675 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:09.400+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:38:09.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:09.674+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:38:10.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:10.260+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:38:10.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:10.414+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:38:10.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:10.654+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:38:11.635 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:11.278+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:38:11.636 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:11.416+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:38:11.636 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:38:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:38:11.351+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 19158 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:38:11.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:11.634+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:38:12.673 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:12.403+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:38:12.674 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:12.295+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:38:12.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:12.673+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:38:13.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:13.265+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:38:13.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:13.362+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:38:14.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:13.705+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:38:14.692 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:14.361+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:38:14.692 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:14.306+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:38:14.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:14.691+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:38:15.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:15.400+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:38:15.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:15.259+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:38:16.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:15.740+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:38:16.697 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:38:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:38:16.353+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 19249 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:38:16.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:16.416+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:38:16.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:16.301+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:38:16.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:16.301+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:38:16.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:16.301+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:38:16.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:16.302+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:38:17.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:16.700+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:38:17.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:17.431+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:38:17.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:17.259+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:38:17.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:17.259+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:38:17.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:17.259+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:38:17.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:17.260+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:38:18.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:17.714+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:38:18.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:18.442+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:38:18.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:18.251+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:38:18.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:18.251+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:38:18.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:18.251+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:38:18.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:18.251+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:38:19.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:18.733+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:38:19.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:19.401+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:38:19.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:19.257+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:38:19.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:19.257+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:38:19.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:19.257+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:38:19.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:19.259+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:38:20.166 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:19.701+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:38:20.432 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:20.264+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:38:20.432 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:20.264+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:38:20.433 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:20.264+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:38:20.433 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:20.265+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:38:20.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:20.680+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:38:20.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:20.431+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:38:21.638 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:38:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:38:21.355+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 19337 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:38:21.639 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:21.431+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:38:21.639 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:21.300+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:38:21.639 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:21.300+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:38:21.639 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:21.300+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:38:21.639 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:21.301+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:38:21.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:21.637+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:38:22.652 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:22.294+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:38:22.652 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:22.294+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:38:22.652 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:22.294+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:38:22.652 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:22.295+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:38:22.652 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:22.412+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:38:22.653 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:22.412+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:38:22.653 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:22.412+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:38:22.653 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:22.413+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:38:22.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:22.682+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:38:23.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:23.249+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:38:23.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:23.249+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:38:23.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:23.249+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:38:23.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:23.250+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:38:23.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:23.411+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:38:23.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:23.411+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:38:23.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:23.411+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:38:23.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:23.413+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:38:24.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:23.731+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:38:24.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:24.288+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:38:24.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:24.288+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:38:24.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:24.288+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:38:24.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:24.289+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:38:24.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:24.395+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:38:24.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:24.395+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:38:24.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:24.395+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:38:24.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:24.396+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:38:25.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:24.701+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:38:25.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:25.356+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:38:25.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:25.356+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:38:25.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:25.356+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:38:25.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:25.357+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:38:25.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:25.290+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:38:25.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:25.290+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:38:25.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:25.290+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:38:25.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:25.291+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:38:26.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:25.710+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:38:26.697 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:38:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:38:26.357+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 19427 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:38:26.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:26.276+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:38:26.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:26.276+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:38:26.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:26.276+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:38:26.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:26.277+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:38:26.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:26.365+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:38:26.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:26.365+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:38:26.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:26.365+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:38:26.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:26.366+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:38:27.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:26.750+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:38:27.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:27.390+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:38:27.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:27.390+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:38:27.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:27.390+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:38:27.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:27.391+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:38:27.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:27.240+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:38:27.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:27.240+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:38:27.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:27.240+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:38:27.699 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:27.241+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:38:28.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:27.705+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:38:28.687 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:28.364+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:38:28.687 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:28.364+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:38:28.687 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:28.364+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:38:28.687 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:28.365+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:38:28.687 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:28.281+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:38:28.688 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:28.281+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:38:28.688 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:28.281+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:38:28.688 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:28.282+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:38:28.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:28.685+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:38:29.681 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:29.286+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:38:29.681 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:29.286+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:38:29.682 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:29.286+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:38:29.682 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:29.287+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:38:29.682 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:29.383+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:38:29.682 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:29.383+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:38:29.682 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:29.383+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:38:29.682 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:29.383+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:38:29.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:29.679+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:38:30.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:30.255+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:38:30.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:30.255+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:38:30.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:30.255+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:38:30.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:30.256+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:38:30.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:30.396+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:38:30.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:30.396+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:38:30.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:30.396+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:38:30.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:30.397+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:38:30.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:30.660+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:38:31.483 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:38:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:38:31.359+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 19515 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:38:31.483 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:31.363+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:38:31.483 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:31.363+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:38:31.483 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:31.363+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:38:31.484 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:31.364+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:38:31.484 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:31.214+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:38:31.484 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:31.214+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:38:31.484 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:31.214+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:38:31.484 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:31.215+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:38:31.867 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:31.667+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:38:31.868 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:31.667+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:38:31.868 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:31.667+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:38:31.868 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:31.668+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:38:32.682 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:32.199+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:38:32.682 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:32.199+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:38:32.682 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:32.199+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:38:32.683 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:32.200+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:38:32.683 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:32.318+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:38:32.683 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:32.318+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:38:32.683 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:32.318+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:38:32.683 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:32.319+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:38:32.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:32.680+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:38:32.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:32.680+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:38:32.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:32.680+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:38:32.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:32.681+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:38:33.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:33.162+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:38:33.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:33.162+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:38:33.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:33.162+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:38:33.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:33.163+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:38:33.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:33.328+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:38:33.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:33.328+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:38:33.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:33.328+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:38:33.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:33.329+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:38:33.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:33.652+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:38:33.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:33.652+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:38:33.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:33.652+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:38:33.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:33.653+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:38:34.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:34.333+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:38:34.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:34.333+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:38:34.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:34.333+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:38:34.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:34.334+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:38:34.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:34.114+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:38:34.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:34.114+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:38:34.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:34.114+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:38:34.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:34.114+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:38:35.153 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:34.697+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:38:35.153 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:34.697+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:38:35.153 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:34.697+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:38:35.154 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:34.698+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:38:35.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:35.151+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:38:35.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:35.151+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:38:35.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:35.151+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:38:35.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:35.152+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:38:35.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:35.375+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:38:35.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:35.375+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:38:35.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:35.375+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:38:35.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:35.375+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:38:35.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:35.671+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:38:35.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:35.671+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:38:35.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:35.671+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:38:35.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:35.672+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:38:36.447 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:38:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:38:36.361+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 19600 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:38:36.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:36.347+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:38:36.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:36.347+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:38:36.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:36.347+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:38:36.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:36.348+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:38:36.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:36.188+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:38:36.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:36.188+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:38:36.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:36.188+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:38:36.449 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:36.189+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:38:36.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:36.706+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:38:36.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:36.706+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:38:36.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:36.706+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:38:36.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:36.707+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:38:37.341 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:37.150+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:38:37.341 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:37.150+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:38:37.341 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:37.150+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:38:37.341 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:37.151+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:38:37.693 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:37.339+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:38:37.693 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:37.339+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:38:37.694 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:37.339+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:38:37.694 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:37.340+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:38:37.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:37.692+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:38:37.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:37.692+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:38:37.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:37.692+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:38:37.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:37.693+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:38:38.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:38.151+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:38:38.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:38.151+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:38:38.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:38.151+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:38:38.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:38.152+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:38:38.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:38.345+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:38:38.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:38.345+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:38:38.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:38.345+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:38:38.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:38.345+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:38:38.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:38.649+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:38:38.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:38.649+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:38:38.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:38.649+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:38:38.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:38.650+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:38:39.389 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:39.115+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:38:39.389 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:39.115+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:38:39.389 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:39.115+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:38:39.389 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:39.116+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:38:39.692 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:39.387+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:38:39.692 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:39.387+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:38:39.692 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:39.387+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:38:39.692 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:39.388+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:38:39.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:39.690+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:38:39.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:39.690+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:38:39.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:39.690+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:38:39.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:39.691+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:38:40.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:40.129+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:38:40.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:40.129+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:38:40.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:40.129+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:38:40.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:40.130+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:38:40.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:40.361+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:38:40.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:40.361+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:38:40.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:40.361+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:38:40.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:40.362+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:38:41.083 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:40.722+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:38:41.084 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:40.722+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:38:41.084 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:40.722+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:38:41.084 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:40.723+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:38:41.337 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:41.135+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:38:41.337 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:41.135+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:38:41.337 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:41.135+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:38:41.338 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:41.136+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:38:41.697 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:38:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:38:41.364+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 19689 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:38:41.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:41.336+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:38:41.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:41.336+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:38:41.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:41.336+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:38:41.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:41.336+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:38:42.002 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:41.740+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:38:42.002 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:41.740+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:38:42.003 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:41.740+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:38:42.003 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:41.741+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:38:42.333 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:42.176+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:38:42.333 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:42.176+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:38:42.333 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:42.176+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:38:42.333 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:42.177+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:38:42.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:42.331+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:38:42.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:42.331+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:38:42.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:42.331+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:38:42.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:42.333+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:38:43.171 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:42.729+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:38:43.171 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:42.729+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:38:43.172 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:42.729+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:38:43.172 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:42.730+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:38:43.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:43.349+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:38:43.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:43.349+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:38:43.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:43.349+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:38:43.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:43.350+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:38:43.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:43.170+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:38:43.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:43.170+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:38:43.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:43.170+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:38:43.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:43.171+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:38:43.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:43.691+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:38:43.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:43.691+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:38:43.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:43.691+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:38:43.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:43.692+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:38:44.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:44.154+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:38:44.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:44.154+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:38:44.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:44.154+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:38:44.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:44.155+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:38:44.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:44.329+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:38:44.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:44.329+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:38:44.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:44.329+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:38:44.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:44.329+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:38:44.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:44.694+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:38:44.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:44.694+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:38:44.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:44.694+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:38:44.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:44.695+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:38:45.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:45.124+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:38:45.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:45.124+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:38:45.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:45.124+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:38:45.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:45.125+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:38:45.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:45.316+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:38:45.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:45.316+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:38:45.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:45.316+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:38:45.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:45.317+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:38:46.115 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:45.727+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:38:46.115 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:45.727+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:38:46.116 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:45.727+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:38:46.116 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:45.728+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:38:46.367 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:46.294+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:38:46.368 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:46.294+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:38:46.368 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:46.294+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:38:46.368 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:46.295+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:38:46.368 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:46.114+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:38:46.368 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:46.114+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:38:46.368 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:46.114+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:38:46.369 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:46.114+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:38:46.697 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:38:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:38:46.366+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 19781 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:38:47.042 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:46.709+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:38:47.043 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:46.709+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:38:47.043 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:46.709+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:38:47.043 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:46.710+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:38:47.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:47.113+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:38:47.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:47.113+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:38:47.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:47.113+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:38:47.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:47.114+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:38:47.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:47.248+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:38:47.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:47.248+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:38:47.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:47.248+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:38:47.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:47.248+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:38:47.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:47.691+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:38:47.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:47.691+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:38:47.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:47.691+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:38:47.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:47.692+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:38:48.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:48.295+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:38:48.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:48.295+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:38:48.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:48.295+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:38:48.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:48.296+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:38:48.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:48.141+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:38:48.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:48.141+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:38:48.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:48.141+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:38:48.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:48.142+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:38:49.099 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:48.698+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:38:49.099 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:48.698+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:38:49.099 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:48.698+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:38:49.099 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:48.699+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:38:49.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:49.097+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:38:49.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:49.097+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:38:49.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:49.097+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:38:49.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:49.098+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:38:49.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:49.342+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:38:49.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:49.342+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:38:49.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:49.342+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:38:49.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:49.343+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:38:49.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:49.693+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:38:49.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:49.693+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:38:49.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:49.693+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:38:49.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:49.694+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:38:50.390 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:50.389+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:38:50.391 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:50.389+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:38:50.391 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:50.139+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:38:50.391 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:50.139+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:38:50.391 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:50.139+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:38:50.391 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:50.140+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:38:50.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:50.389+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:38:50.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:50.389+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:38:51.152 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:50.733+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:38:51.152 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:50.733+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:38:51.153 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:50.733+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:38:51.153 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:50.734+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:38:51.447 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:38:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:38:51.368+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 19870 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:38:51.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:51.151+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:38:51.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:51.151+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:38:51.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:51.151+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:38:51.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:51.152+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:38:51.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:51.345+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:38:51.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:51.345+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:38:51.449 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:51.345+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:38:51.449 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:51.346+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:38:52.082 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:51.769+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:38:52.082 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:51.769+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:38:52.083 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:51.769+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:38:52.083 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:51.770+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:38:52.337 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:52.109+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:38:52.337 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:52.109+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:38:52.337 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:52.109+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:38:52.337 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:52.110+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:38:52.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:52.335+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:38:52.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:52.335+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:38:52.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:52.335+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:38:52.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:52.336+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:38:53.106 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:52.763+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:38:53.106 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:52.763+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:38:53.107 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:52.763+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:38:53.107 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:52.764+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:38:53.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:53.105+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:38:53.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:53.105+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:38:53.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:53.105+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:38:53.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:53.106+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:38:53.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:53.342+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:38:53.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:53.342+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:38:53.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:53.342+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:38:53.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:53.343+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:38:54.059 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:53.802+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:38:54.059 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:53.802+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:38:54.060 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:53.802+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:38:54.060 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:53.803+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:38:54.314 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:54.058+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:38:54.314 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:54.058+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:38:54.314 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:54.058+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:38:54.314 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:54.059+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:38:54.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:54.312+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:38:54.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:54.312+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:38:54.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:54.312+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:38:54.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:54.313+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:38:55.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:54.798+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:38:55.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:54.798+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:38:55.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:54.798+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:38:55.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:54.799+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:38:55.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:55.030+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:38:55.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:55.030+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:38:55.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:55.030+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:38:55.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:55.031+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:38:55.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:55.350+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:38:55.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:55.350+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:38:55.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:55.350+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:38:55.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:55.351+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:38:56.053 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:55.798+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:38:56.053 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:55.798+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:38:56.053 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:55.798+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:38:56.053 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:55.799+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:38:56.371 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:56.051+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:38:56.372 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:56.051+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:38:56.372 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:56.051+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:38:56.372 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:56.052+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:38:56.697 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:38:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:38:56.370+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 19963 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:38:56.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:56.370+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:38:56.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:56.370+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:38:56.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:56.370+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:38:56.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:56.372+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:38:57.124 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:57.030+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:38:57.124 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:57.030+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:38:57.124 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:57.030+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:38:57.124 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:57.031+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:38:57.124 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:56.827+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:38:57.124 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:56.827+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:38:57.125 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:56.827+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:38:57.125 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:56.828+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:38:57.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:57.352+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:38:57.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:57.352+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:38:57.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:57.352+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:38:57.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:57.353+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:38:58.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:57.820+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:38:58.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:57.820+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:38:58.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:57.820+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:38:58.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:57.821+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:38:58.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:58.039+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:38:58.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:58.039+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:38:58.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:58.039+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:38:58.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:58.041+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:38:58.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:58.338+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:38:58.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:58.338+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:38:58.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:58.338+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:38:58.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:58.339+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:38:59.050 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:58.796+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:38:59.050 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:58.796+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:38:59.050 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:58.796+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:38:59.050 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:58.797+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:38:59.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:59.049+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:38:59.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:59.049+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:38:59.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:59.049+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:38:59.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:38:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:38:59.050+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:38:59.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:59.292+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:38:59.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:59.292+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:38:59.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:59.292+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:38:59.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:38:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:38:59.293+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:39:00.086 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:59.801+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:00.086 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:59.801+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:00.086 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:59.801+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:00.087 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:38:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:38:59.801+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:39:00.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:00.085+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:00.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:00.085+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:00.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:00.085+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:00.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:00.086+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:39:00.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:00.247+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:00.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:00.247+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:00.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:00.247+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:00.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:00.248+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:39:01.131 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:00.786+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:01.132 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:00.786+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:01.132 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:00.786+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:01.132 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:00.787+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:39:01.386 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:39:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:39:01.373+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 20052 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:39:01.387 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:01.286+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:01.387 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:01.286+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:01.387 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:01.286+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:01.387 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:01.287+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:39:01.387 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:01.131+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:01.388 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:01.131+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:01.388 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:01.131+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:01.388 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:01.132+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:39:02.145 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:01.771+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:02.145 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:01.771+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:02.145 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:01.771+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:02.145 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:01.771+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:39:02.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:02.143+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:02.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:02.143+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:02.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:02.143+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:02.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:02.144+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:39:02.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:02.313+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:02.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:02.313+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:02.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:02.313+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:02.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:02.313+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:39:02.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:02.793+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:02.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:02.793+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:02.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:02.793+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:02.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:02.794+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:39:03.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:03.101+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:03.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:03.101+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:03.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:03.101+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:03.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:03.102+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:39:03.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:03.302+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:03.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:03.302+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:03.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:03.302+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:03.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:03.303+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:39:04.094 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:03.830+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:04.094 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:03.830+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:04.094 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:03.830+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:04.095 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:03.831+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:39:04.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:04.285+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:04.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:04.285+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:04.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:04.285+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:04.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:04.286+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:39:04.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:04.092+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:04.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:04.092+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:04.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:04.092+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:04.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:04.094+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:39:05.125 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:04.834+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:05.126 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:04.834+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:05.126 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:04.834+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:05.126 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:04.835+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:39:05.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:05.124+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:05.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:05.124+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:05.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:05.124+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:05.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:05.125+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:39:05.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:05.323+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:05.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:05.323+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:05.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:05.323+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:05.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:05.323+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:39:06.150 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:05.840+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:06.150 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:05.840+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:06.150 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:05.840+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:06.151 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:05.841+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:39:06.447 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:39:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:39:06.374+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 20144 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:39:06.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:06.368+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:06.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:06.368+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:06.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:06.368+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:06.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:06.368+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:39:06.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:06.148+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:06.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:06.148+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:06.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:06.148+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:06.449 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:06.149+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:39:07.134 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:06.825+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:07.134 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:06.825+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:07.134 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:06.825+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:07.134 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:06.826+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:39:07.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:07.132+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:07.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:07.132+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:07.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:07.132+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:07.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:07.133+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:39:07.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:07.324+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:07.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:07.324+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:07.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:07.324+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:07.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:07.325+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:39:08.130 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:07.816+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:08.131 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:07.816+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:08.131 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:07.816+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:08.131 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:07.817+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:39:08.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:08.128+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:08.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:08.129+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:08.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:08.129+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:08.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:08.130+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:39:08.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:08.322+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:08.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:08.322+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:08.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:08.322+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:08.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:08.323+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:39:09.138 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:08.802+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:09.141 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:08.802+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:09.142 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:08.802+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:09.142 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:08.802+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:39:09.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:09.350+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:09.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:09.350+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:09.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:09.350+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:09.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:09.350+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:39:09.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:09.136+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:09.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:09.136+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:09.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:09.136+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:09.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:09.138+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:39:10.117 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:09.787+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:10.117 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:09.787+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:10.117 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:09.787+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:10.117 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:09.788+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:39:10.371 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:10.115+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:10.371 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:10.115+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:10.371 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:10.115+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:10.371 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:10.116+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:39:10.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:10.369+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:10.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:10.369+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:10.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:10.369+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:10.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:10.370+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:39:11.135 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:10.806+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:11.135 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:10.806+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:11.135 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:10.806+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:11.135 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:10.807+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:39:11.391 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:11.133+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:11.391 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:11.133+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:11.391 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:11.133+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:11.392 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:11.134+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:39:11.392 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:39:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:39:11.377+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 20233 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:39:11.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:11.400+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:11.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:11.400+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:11.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:11.400+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:11.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:11.401+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:39:12.124 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:11.820+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:12.124 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:11.820+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:12.124 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:11.820+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:12.124 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:11.821+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:39:12.445 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:12.122+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:12.446 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:12.122+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:12.446 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:12.122+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:12.446 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:12.123+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:39:12.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:12.444+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:12.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:12.444+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:12.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:12.444+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:12.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:12.445+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:39:13.089 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:12.824+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:13.090 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:12.824+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:13.090 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:12.824+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:13.090 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:12.825+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:39:13.404 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:13.088+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:13.404 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:13.088+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:13.404 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:13.088+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:13.404 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:13.089+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:39:13.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:13.402+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:13.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:13.402+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:13.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:13.402+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:13.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:13.403+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:39:14.125 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:13.795+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:14.125 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:13.795+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:14.125 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:13.795+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:14.125 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:13.796+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:39:14.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:14.123+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:14.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:14.123+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:14.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:14.123+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:14.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:14.124+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:39:14.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:14.356+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:14.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:14.356+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:14.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:14.356+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:14.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:14.357+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:39:15.164 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:14.748+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:15.165 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:14.748+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:15.165 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:14.748+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:15.165 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:14.748+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:39:15.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:15.163+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:15.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:15.163+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:15.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:15.163+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:15.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:15.164+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:39:15.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:15.331+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:15.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:15.331+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:15.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:15.331+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:15.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:15.332+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:39:16.138 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:15.745+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:16.138 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:15.745+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:16.138 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:15.745+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:16.138 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:15.745+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:39:16.399 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:39:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:39:16.379+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 20326 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:39:16.400 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:16.136+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:16.400 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:16.136+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:16.400 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:16.136+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:16.400 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:16.137+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:39:16.401 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:16.360+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:16.401 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:16.360+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:16.401 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:16.360+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:16.401 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:16.361+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:39:17.117 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:16.703+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:17.118 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:16.703+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:17.118 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:16.703+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:17.118 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:16.703+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:39:17.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:17.114+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:17.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:17.114+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:17.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:17.114+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:17.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:17.115+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:39:17.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:17.363+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:17.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:17.363+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:17.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:17.363+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:17.449 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:17.363+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:39:18.164 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:17.700+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:18.164 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:17.700+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:18.165 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:17.700+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:18.165 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:17.701+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:39:18.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:18.162+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:18.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:18.162+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:18.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:18.162+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:18.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:18.163+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:39:18.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:18.352+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:18.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:18.352+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:18.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:18.352+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:18.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:18.353+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:39:18.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:18.652+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:18.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:18.652+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:18.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:18.652+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:18.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:18.653+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:39:19.676 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:19.197+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:19.676 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:19.197+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:19.676 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:19.197+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:19.676 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:19.198+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:39:19.677 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:19.354+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:19.677 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:19.354+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:19.677 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:19.354+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:19.677 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:19.355+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:39:19.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:19.674+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:19.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:19.674+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:19.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:19.674+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:19.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:19.675+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:39:20.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:20.307+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:20.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:20.307+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:20.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:20.307+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:20.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:20.307+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:39:20.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:20.245+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:20.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:20.245+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:20.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:20.245+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:20.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:20.246+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:39:20.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:20.679+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:20.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:20.679+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:20.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:20.679+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:20.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:20.681+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:39:21.330 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:21.259+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:21.330 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:21.259+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:21.331 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:21.259+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:21.331 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:21.260+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:39:21.697 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:39:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:39:21.381+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 20415 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:39:21.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:21.329+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:21.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:21.329+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:21.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:21.329+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:21.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:21.329+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:39:22.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:21.724+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:22.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:21.724+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:22.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:21.724+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:22.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:21.725+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:39:22.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:22.242+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:22.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:22.242+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:22.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:22.242+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:22.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:22.243+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:39:22.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:22.378+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:22.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:22.378+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:22.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:22.378+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:22.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:22.379+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:39:23.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:22.738+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:23.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:22.738+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:23.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:22.738+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:23.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:22.739+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:39:23.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:23.197+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:23.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:23.197+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:23.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:23.197+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:23.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:23.198+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:39:23.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:23.382+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:23.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:23.383+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:23.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:23.383+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:23.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:23.383+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:39:24.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:23.707+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:24.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:23.707+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:24.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:23.707+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:24.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:23.708+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:39:24.694 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:24.197+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:24.694 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:24.197+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:24.694 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:24.197+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:24.694 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:24.198+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:39:24.695 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:24.388+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:24.695 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:24.388+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:24.695 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:24.388+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:24.695 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:24.389+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:39:24.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:24.691+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:24.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:24.691+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:24.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:24.691+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:24.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:24.691+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:39:25.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:25.382+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:25.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:25.382+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:25.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:25.382+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:25.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:25.382+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:39:25.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:25.166+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:25.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:25.166+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:25.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:25.166+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:25.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:25.167+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:39:26.143 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:25.738+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:26.143 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:25.738+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:26.143 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:25.738+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:26.144 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:25.739+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:39:26.396 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:39:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:39:26.383+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 20507 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:39:26.397 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:26.141+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:26.397 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:26.141+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:26.397 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:26.141+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:26.397 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:26.142+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:39:26.397 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:26.345+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:26.398 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:26.345+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:26.398 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:26.345+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:26.398 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:26.346+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:39:27.125 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:26.769+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:27.125 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:26.769+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:27.126 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:26.769+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:27.126 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:26.770+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:39:27.389 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:27.124+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:27.390 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:27.124+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:27.390 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:27.124+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:27.390 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:27.125+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:39:27.390 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:27.360+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:27.390 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:27.360+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:27.391 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:27.360+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:27.391 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:27.361+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:39:28.145 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:27.740+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:28.145 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:27.740+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:28.145 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:27.740+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:28.145 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:27.741+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:39:28.412 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:28.143+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:28.413 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:28.143+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:28.413 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:28.143+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:28.413 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:28.144+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:39:28.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:28.411+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:28.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:28.411+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:28.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:28.411+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:28.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:28.411+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:39:29.140 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:28.775+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:29.141 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:28.775+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:29.141 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:28.775+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:29.141 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:28.776+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:39:29.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:29.139+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:29.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:29.139+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:29.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:29.139+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:29.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:29.140+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:39:29.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:29.371+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:29.449 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:29.371+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:29.449 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:29.371+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:29.449 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:29.372+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:39:30.114 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:29.732+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:30.114 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:29.732+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:30.114 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:29.732+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:30.114 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:29.733+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:39:30.377 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:30.112+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:30.378 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:30.112+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:30.378 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:30.112+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:30.378 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:30.113+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:39:30.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:30.376+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:30.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:30.376+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:30.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:30.376+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:30.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:30.377+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:39:31.107 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:30.714+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:31.107 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:30.714+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:31.107 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:30.714+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:31.108 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:30.714+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:39:31.389 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:31.105+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:31.389 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:31.105+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:31.389 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:31.105+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:31.390 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:31.106+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:39:31.679 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:39:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:39:31.385+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 20596 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:39:31.680 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:31.421+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:31.680 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:31.421+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:31.680 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:31.421+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:31.680 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:31.422+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:39:31.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:31.678+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:31.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:31.678+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:31.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:31.678+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:31.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:31.679+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:39:32.396 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:32.070+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:32.396 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:32.070+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:32.396 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:32.070+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:32.396 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:32.071+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:39:32.656 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:32.394+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:32.656 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:32.394+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:32.656 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:32.394+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:32.657 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:32.395+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:39:32.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:32.654+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:32.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:32.654+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:32.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:32.654+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:32.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:32.655+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:39:33.430 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:33.085+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:33.430 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:33.085+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:33.430 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:33.085+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:33.430 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:33.086+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:39:33.696 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:33.428+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:33.696 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:33.428+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:33.696 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:33.428+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:33.696 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:33.429+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:39:33.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:33.694+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:33.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:33.694+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:33.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:33.694+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:33.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:33.695+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:39:34.382 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:34.107+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:34.382 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:34.107+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:34.382 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:34.107+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:34.382 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:34.108+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:39:34.673 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:34.380+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:34.673 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:34.380+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:34.673 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:34.380+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:34.674 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:34.381+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:39:34.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:34.671+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:34.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:34.671+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:34.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:34.671+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:34.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:34.672+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:39:35.421 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:35.092+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:35.421 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:35.092+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:35.421 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:35.092+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:35.422 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:35.093+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:39:35.683 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:35.419+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:35.683 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:35.419+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:35.683 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:35.419+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:35.684 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:35.420+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:39:35.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:35.682+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:35.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:35.682+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:35.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:35.682+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:35.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:35.682+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:39:36.389 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:36.111+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:36.389 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:36.111+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:36.390 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:36.111+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:36.390 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:36.112+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:39:36.697 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:39:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:39:36.388+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 20689 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:39:36.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:36.448+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:36.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:36.448+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:36.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:36.448+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:36.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:36.449+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:39:37.115 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:36.714+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:37.115 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:36.714+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:37.115 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:36.714+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:37.116 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:36.715+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:39:37.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:37.113+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:37.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:37.113+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:37.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:37.113+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:37.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:37.114+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:39:37.710 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:37.708+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:37.711 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:37.708+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:37.711 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:37.708+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:37.711 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:37.709+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:39:37.711 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:37.458+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:37.711 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:37.458+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:37.712 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:37.458+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:37.712 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:37.459+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:39:38.417 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:38.086+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:38.417 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:38.086+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:38.417 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:38.086+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:38.417 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:38.087+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:39:38.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:38.415+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:38.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:38.415+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:38.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:38.415+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:38.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:38.416+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:39:38.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:38.695+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:38.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:38.695+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:38.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:38.695+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:38.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:38.696+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:39:39.410 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:39.121+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:39.410 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:39.121+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:39.410 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:39.121+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:39.410 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:39.122+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:39:39.686 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:39.408+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:39.687 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:39.408+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:39.687 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:39.408+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:39.687 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:39.409+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:39:39.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:39.684+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:39.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:39.684+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:39.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:39.684+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:39.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:39.685+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:39:40.424 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:40.152+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:40.424 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:40.152+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:40.425 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:40.152+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:40.425 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:40.153+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:39:40.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:40.423+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:40.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:40.423+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:40.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:40.423+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:40.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:40.424+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:39:40.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:40.652+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:40.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:40.652+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:40.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:40.652+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:40.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:40.653+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:39:41.413 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:39:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:39:41.390+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 20778 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:39:41.414 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:41.161+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:41.414 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:41.161+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:41.414 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:41.161+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:41.414 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:41.162+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:39:41.415 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:41.406+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:41.415 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:41.406+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:41.415 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:41.406+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:41.415 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:41.407+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:39:41.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:41.627+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:41.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:41.627+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:41.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:41.627+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:41.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:41.628+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:39:42.435 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:42.134+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:42.435 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:42.134+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:42.435 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:42.134+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:42.435 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:42.134+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:39:42.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:42.638+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:42.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:42.638+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:42.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:42.638+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:42.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:42.638+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:39:42.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:42.434+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:42.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:42.434+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:42.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:42.434+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:42.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:42.434+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:39:43.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:43.423+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:43.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:43.423+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:43.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:43.423+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:43.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:43.424+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:39:43.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:43.182+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:43.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:43.182+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:43.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:43.182+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:43.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:43.183+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:39:43.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:43.637+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:43.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:43.637+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:43.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:43.637+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:43.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:43.638+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:39:44.686 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:44.473+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:44.686 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:44.474+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:44.686 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:44.474+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:44.686 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:44.474+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:39:44.687 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:44.227+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:44.687 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:44.227+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:44.687 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:44.227+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:44.687 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:44.228+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:39:44.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:44.685+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:44.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:44.685+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:44.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:44.685+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:44.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:44.686+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:39:45.646 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:45.247+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:45.647 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:45.247+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:45.647 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:45.247+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:45.647 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:45.248+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:39:45.647 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:45.467+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:45.647 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:45.467+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:45.648 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:45.467+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:45.648 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:45.468+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:39:45.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:45.646+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:45.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:45.646+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:45.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:45.646+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:45.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:45.647+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:39:46.472 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:39:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:39:46.393+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 20870 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:39:46.472 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:46.218+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:46.472 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:46.218+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:46.473 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:46.218+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:46.473 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:46.219+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:39:46.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:46.621+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:46.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:46.621+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:46.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:46.621+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:46.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:46.621+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:39:46.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:46.472+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:46.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:46.472+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:46.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:46.472+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:46.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:46.473+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:39:47.567 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:47.438+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:47.568 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:47.438+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:47.568 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:47.438+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:47.568 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:47.438+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:39:47.568 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:47.251+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:47.568 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:47.251+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:47.569 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:47.251+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:47.569 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:47.252+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:39:47.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:47.584+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:47.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:47.584+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:47.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:47.584+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:47.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:47.584+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:39:48.628 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:48.285+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:48.628 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:48.285+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:48.629 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:48.285+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:48.629 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:48.286+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:39:48.629 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:48.407+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:48.629 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:48.407+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:48.629 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:48.407+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:48.629 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:48.408+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:39:48.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:48.626+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:48.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:48.627+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:48.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:48.627+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:48.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:48.627+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:39:49.652 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:49.396+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:49.652 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:49.396+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:49.652 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:49.396+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:49.653 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:49.397+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:39:49.653 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:49.320+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:49.653 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:49.320+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:49.653 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:49.320+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:49.653 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:49.321+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:39:49.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:49.651+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:49.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:49.651+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:49.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:49.651+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:49.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:49.652+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:39:50.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:50.313+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:50.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:50.313+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:50.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:50.313+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:50.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:50.313+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:39:50.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:50.420+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:50.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:50.420+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:50.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:50.420+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:50.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:50.421+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:39:50.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:50.604+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:50.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:50.604+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:50.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:50.604+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:50.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:50.605+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:39:51.697 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:39:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:39:51.394+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 20957 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:39:51.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:51.586+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:51.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:51.586+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:51.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:51.586+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:51.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:51.587+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:39:51.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:51.467+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:51.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:51.467+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:51.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:51.467+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:51.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:51.467+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:39:51.699 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:51.344+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:51.699 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:51.344+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:51.699 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:51.344+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:51.699 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:51.345+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:39:52.620 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:52.343+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:52.621 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:52.343+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:52.621 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:52.343+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:52.621 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:52.344+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:39:52.621 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:52.547+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:52.621 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:52.547+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:52.622 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:52.547+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:52.622 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:52.548+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:39:52.622 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:52.457+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:52.622 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:52.457+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:52.622 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:52.457+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:52.622 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:52.458+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:39:53.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:53.590+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:53.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:53.590+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:53.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:53.590+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:53.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:53.591+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:39:53.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:53.350+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:53.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:53.350+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:53.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:53.350+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:53.699 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:53.351+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:39:53.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:53.445+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:53.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:53.445+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:53.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:53.445+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:53.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:53.446+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:39:54.583 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:54.323+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:54.583 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:54.323+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:54.584 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:54.323+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:54.584 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:54.323+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:39:54.584 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:54.451+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:54.584 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:54.451+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:54.584 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:54.451+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:54.584 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:54.451+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:39:54.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:54.581+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:54.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:54.581+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:54.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:54.581+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:54.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:54.582+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:39:55.608 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:55.304+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:55.608 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:55.304+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:55.608 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:55.304+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:55.608 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:55.305+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:39:55.609 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:55.417+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:55.609 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:55.417+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:55.609 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:55.417+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:55.609 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:55.418+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:39:55.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:55.606+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:55.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:55.606+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:55.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:55.606+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:55.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:55.607+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:39:56.601 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:39:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:39:56.397+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 21046 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:39:56.602 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:56.449+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:56.602 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:56.449+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:56.602 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:56.449+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:56.602 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:56.450+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:39:56.602 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:56.297+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:56.603 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:56.297+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:56.603 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:56.297+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:56.603 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:56.298+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:39:56.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:56.600+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:56.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:56.600+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:56.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:56.600+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:56.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:56.600+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:39:57.659 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:57.571+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:57.660 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:57.571+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:57.660 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:57.571+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:57.660 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:57.572+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:39:57.660 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:57.421+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:57.660 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:57.421+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:57.661 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:57.421+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:57.661 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:57.422+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:39:57.661 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:57.332+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:57.661 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:57.332+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:57.661 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:57.332+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:57.661 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:57.333+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:39:58.582 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:58.299+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:58.582 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:58.299+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:58.582 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:58.299+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:58.582 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:58.299+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:39:58.583 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:58.416+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:58.583 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:58.416+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:58.583 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:58.416+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:58.583 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:58.417+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:39:58.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:58.580+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:58.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:58.580+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:58.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:58.580+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:58.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:58.581+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:39:59.630 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:59.432+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:59.630 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:59.432+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:59.630 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:59.432+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:39:59.631 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:39:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:39:59.433+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:39:59.631 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:59.338+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:59.631 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:59.338+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:59.631 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:59.338+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:39:59.631 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:39:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:39:59.340+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:39:59.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:59.628+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:59.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:59.628+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:59.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:59.628+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:39:59.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:39:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:39:59.629+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:40:00.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:00.313+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:00.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:00.313+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:00.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:00.313+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:00.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:00.314+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:40:00.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:00.625+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:00.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:00.625+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:00.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:00.625+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:00.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:00.626+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:40:00.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:00.465+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:00.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:00.465+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:00.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:00.465+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:00.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:00.466+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:40:01.401 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:01.355+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:01.401 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:01.355+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:01.402 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:01.355+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:01.402 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:01.356+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:40:01.697 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:40:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:40:01.399+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 21130 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:40:01.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:01.617+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:01.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:01.617+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:01.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:01.617+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:01.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:01.618+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:40:01.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:01.419+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:01.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:01.419+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:01.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:01.419+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:01.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:01.420+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:40:02.609 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:02.329+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:02.609 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:02.329+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:02.609 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:02.329+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:02.610 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:02.330+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:40:02.610 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:02.443+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:02.610 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:02.443+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:02.610 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:02.443+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:02.610 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:02.444+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:40:02.611 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:02.607+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:02.611 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:02.607+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:02.611 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:02.607+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:02.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:02.608+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:40:03.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:03.341+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:03.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:03.341+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:03.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:03.341+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:03.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:03.342+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:40:03.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:03.570+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:03.699 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:03.570+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:03.699 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:03.570+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:03.699 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:03.570+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:40:03.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:03.403+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:03.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:03.403+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:03.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:03.403+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:03.700 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:03.405+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:40:04.599 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:04.372+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:04.600 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:04.372+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:04.600 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:04.372+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:04.600 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:04.372+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:40:04.600 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:04.534+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:04.601 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:04.534+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:04.601 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:04.534+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:04.601 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:04.535+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:40:04.601 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:04.315+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:04.601 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:04.315+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:04.601 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:04.315+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:04.602 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:04.316+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:40:05.552 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:05.388+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:05.552 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:05.388+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:05.552 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:05.388+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:05.552 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:05.389+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:40:05.553 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:05.283+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:05.553 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:05.283+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:05.553 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:05.283+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:05.553 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:05.284+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:40:05.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:05.550+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:05.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:05.550+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:05.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:05.550+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:05.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:05.551+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:40:06.697 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:40:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:40:06.401+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 21216 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:40:06.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:06.406+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:06.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:06.406+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:06.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:06.406+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:06.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:06.407+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:40:06.699 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:06.569+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:06.699 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:06.569+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:06.699 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:06.569+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:06.699 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:06.569+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:40:06.700 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:06.325+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:06.700 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:06.325+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:06.700 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:06.325+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:06.700 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:06.325+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:40:07.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:07.566+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:07.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:07.566+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:07.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:07.566+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:07.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:07.566+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:40:07.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:07.379+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:07.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:07.379+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:07.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:07.379+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:07.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:07.379+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:40:07.699 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:07.346+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:07.699 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:07.346+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:07.699 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:07.346+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:07.699 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:07.347+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:40:08.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:08.553+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:08.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:08.553+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:08.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:08.553+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:08.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:08.553+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:40:08.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:08.380+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:08.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:08.380+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:08.699 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:08.380+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:08.699 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:08.381+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:40:08.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:08.391+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:08.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:08.391+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:08.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:08.391+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:08.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:08.392+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:40:09.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:09.402+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:09.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:09.402+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:09.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:09.402+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:09.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:09.402+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:40:09.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:09.399+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:09.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:09.399+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:09.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:09.399+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:09.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:09.400+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:40:09.699 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:09.602+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:09.699 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:09.602+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:09.700 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:09.602+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:09.700 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:09.603+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:40:10.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:10.385+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:10.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:10.385+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:10.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:10.385+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:10.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:10.386+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:40:10.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:10.383+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:10.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:10.383+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:10.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:10.383+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:10.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:10.384+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:40:10.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:10.640+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:10.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:10.640+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:10.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:10.640+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:10.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:10.641+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:40:11.697 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:40:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:40:11.403+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 21296 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:40:11.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:11.412+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:11.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:11.412+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:11.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:11.412+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:11.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:11.413+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:40:11.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:11.634+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:11.699 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:11.634+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:11.699 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:11.634+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:11.699 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:11.635+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:40:11.699 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:11.419+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:11.699 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:11.419+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:11.699 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:11.419+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:11.699 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:11.420+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:40:12.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:12.396+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:12.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:12.396+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:12.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:12.396+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:12.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:12.397+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:40:12.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:12.414+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:12.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:12.414+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:12.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:12.414+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:12.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:12.415+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:40:12.699 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:12.642+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:12.699 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:12.642+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:12.699 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:12.642+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:12.699 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:12.643+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:40:13.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:13.616+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:13.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:13.616+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:13.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:13.616+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:13.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:13.617+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:40:13.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:13.407+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:13.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:13.407+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:13.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:13.407+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:13.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:13.408+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:40:13.699 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:13.398+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:13.699 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:13.398+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:13.699 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:13.398+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:13.699 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:13.399+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:40:14.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:14.610+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:14.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:14.610+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:14.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:14.610+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:14.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:14.610+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:40:14.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:14.449+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:14.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:14.449+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:14.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:14.449+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:14.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:14.450+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:40:14.699 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:14.367+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:14.699 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:14.367+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:14.699 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:14.367+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:14.699 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:14.367+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:40:15.627 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:15.499+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:15.627 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:15.499+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:15.627 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:15.499+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:15.627 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:15.499+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:40:15.628 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:15.340+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:15.628 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:15.340+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:15.628 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:15.340+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:15.628 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:15.341+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:40:15.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:15.625+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:15.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:15.625+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:15.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:15.625+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:15.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:15.626+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:40:16.655 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:16.495+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:16.656 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:16.495+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:16.656 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:16.495+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:16.656 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:16.497+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:40:16.657 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:16.307+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:16.657 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:16.307+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:16.657 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:16.307+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:16.657 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:16.307+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:40:16.658 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:40:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:40:16.406+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 21383 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:40:16.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:16.654+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:16.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:16.654+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:16.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:16.654+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:16.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:16.655+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:40:17.617 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:17.291+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:17.618 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:17.291+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:17.618 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:17.291+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:17.618 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:17.292+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:40:17.618 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:17.475+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:17.619 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:17.475+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:17.619 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:17.475+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:17.619 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:17.476+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:40:17.619 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:17.616+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:17.619 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:17.616+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:17.619 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:17.616+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:17.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:17.617+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:40:18.605 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:18.294+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:18.605 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:18.294+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:18.605 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:18.294+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:18.606 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:18.294+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:40:18.606 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:18.515+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:18.606 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:18.515+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:18.607 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:18.515+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:18.607 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:18.515+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:40:18.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:18.603+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:18.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:18.603+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:18.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:18.603+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:18.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:18.603+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:40:19.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:19.577+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:19.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:19.577+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:19.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:19.577+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:19.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:19.577+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:40:19.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:19.344+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:19.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:19.344+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:19.699 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:19.344+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:19.699 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:19.345+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:40:19.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:19.520+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:19.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:19.520+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:19.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:19.520+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:19.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:19.521+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:40:20.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:20.370+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:20.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:20.370+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:20.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:20.370+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:20.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:20.370+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:40:20.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:20.533+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:20.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:20.533+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:20.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:20.533+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:20.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:20.534+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:40:20.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:20.578+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:20.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:20.578+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:20.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:20.578+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:20.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:20.579+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:40:21.697 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:40:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:40:21.408+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 21466 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:40:21.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:21.546+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:21.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:21.546+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:21.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:21.546+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:21.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:21.546+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:40:21.699 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:21.355+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:21.699 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:21.355+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:21.699 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:21.355+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:21.699 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:21.355+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:40:21.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:21.584+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:21.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:21.584+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:21.700 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:21.584+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:21.700 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:21.585+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:40:22.651 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:22.631+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:22.652 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:22.631+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:22.652 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:22.631+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:22.653 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:22.632+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:40:22.653 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:22.388+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:22.653 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:22.388+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:22.654 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:22.388+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:22.654 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:22.389+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:40:22.654 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:22.499+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:22.655 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:22.499+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:22.655 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:22.499+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:22.655 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:22.500+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:40:23.624 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:23.518+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:23.624 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:23.518+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:23.625 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:23.518+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:23.625 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:23.519+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:40:23.625 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:23.344+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:23.625 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:23.344+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:23.625 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:23.344+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:23.625 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:23.345+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:40:23.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:23.623+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:23.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:23.623+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:23.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:23.623+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:23.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:23.623+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:40:24.614 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:24.470+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:24.615 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:24.470+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:24.615 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:24.470+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:24.615 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:24.470+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:40:24.615 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:24.363+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:24.616 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:24.363+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:24.616 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:24.363+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:24.616 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:24.363+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:40:24.616 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:24.612+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:24.616 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:24.612+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:24.616 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:24.612+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:24.617 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:24.613+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:40:25.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:25.373+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:25.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:25.373+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:25.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:25.373+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:25.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:25.374+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:40:25.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:25.498+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:25.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:25.498+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:25.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:25.498+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:25.699 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:25.499+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:40:25.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:25.609+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:25.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:25.609+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:25.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:25.609+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:25.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:25.610+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:40:26.660 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:40:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:40:26.410+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 21555 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:40:26.661 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:26.465+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:26.661 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:26.465+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:26.661 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:26.465+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:26.661 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:26.466+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:40:26.661 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:26.375+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:26.662 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:26.375+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:26.662 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:26.375+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:26.662 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:26.376+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:40:26.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:26.659+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:26.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:26.659+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:26.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:26.659+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:26.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:26.660+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:40:27.660 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:27.459+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:27.661 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:27.459+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:27.661 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:27.459+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:27.661 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:27.460+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:40:27.661 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:27.333+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:27.661 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:27.333+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:27.662 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:27.333+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:27.662 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:27.334+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:40:27.933 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:27.659+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:27.933 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:27.659+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:27.933 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:27.659+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:27.933 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:27.660+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:40:28.657 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:28.507+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:28.657 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:28.507+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:28.658 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:28.507+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:28.658 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:28.508+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:40:28.658 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:28.375+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:28.658 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:28.375+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:28.658 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:28.375+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:28.658 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:28.376+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:40:28.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:28.656+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:28.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:28.656+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:28.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:28.656+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:28.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:28.657+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:40:29.658 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:29.532+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:29.658 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:29.532+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:29.659 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:29.532+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:29.659 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:29.532+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:40:29.659 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:29.362+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:29.659 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:29.362+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:29.659 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:29.362+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:29.659 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:29.362+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:40:29.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:29.658+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:29.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:29.658+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:29.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:29.658+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:29.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:29.658+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:40:30.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:30.353+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:30.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:30.353+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:30.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:30.353+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:30.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:30.354+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:40:30.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:30.571+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:30.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:30.571+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:30.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:30.571+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:30.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:30.571+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:40:30.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:30.638+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:30.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:30.638+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:30.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:30.638+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:30.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:30.638+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:40:31.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:31.529+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:31.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:31.529+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:31.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:31.529+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:31.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:31.530+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:40:31.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:31.598+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:31.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:31.598+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:31.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:31.598+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:31.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:31.599+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:40:31.699 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:31.354+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:31.699 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:31.354+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:31.700 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:31.354+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:31.700 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:31.355+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:40:31.700 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:40:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:40:31.414+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 21641 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:40:32.570 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:32.314+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:32.571 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:32.314+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:32.571 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:32.314+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:32.571 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:32.315+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:40:32.571 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:32.551+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:32.571 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:32.551+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:32.572 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:32.551+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:32.572 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:32.552+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:40:32.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:32.570+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:32.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:32.570+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:32.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:32.570+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:32.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:32.571+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:40:33.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:33.576+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:33.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:33.576+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:33.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:33.576+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:33.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:33.577+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:40:33.699 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:33.336+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:33.699 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:33.336+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:33.699 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:33.336+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:33.699 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:33.337+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:40:33.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:33.532+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:33.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:33.532+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:33.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:33.532+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:33.700 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:33.533+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:40:34.575 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:34.313+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:34.576 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:34.313+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:34.576 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:34.313+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:34.576 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:34.313+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:40:34.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:34.574+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:34.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:34.574+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:34.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:34.574+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:34.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:34.575+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:40:34.949 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:34.576+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:34.949 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:34.576+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:34.949 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:34.576+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:34.949 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:34.577+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:40:35.595 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:35.331+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:35.595 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:35.331+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:35.595 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:35.331+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:35.595 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:35.331+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:40:35.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:35.593+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:35.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:35.593+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:35.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:35.593+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:35.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:35.594+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:40:35.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:35.624+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:35.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:35.624+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:35.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:35.624+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:35.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:35.624+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:40:36.627 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:40:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:40:36.415+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 21733 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:40:36.627 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:36.371+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:36.627 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:36.371+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:36.627 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:36.371+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:36.628 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:36.372+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:40:36.628 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:36.586+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:36.628 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:36.586+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:36.628 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:36.586+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:36.628 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:36.587+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:40:36.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:36.625+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:36.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:36.625+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:36.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:36.625+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:36.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:36.626+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:40:37.586 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:37.584+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:37.586 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:37.584+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:37.586 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:37.584+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:37.587 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:37.334+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:37.587 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:37.334+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:37.587 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:37.334+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:37.587 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:37.335+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:40:37.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:37.585+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:40:37.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:37.635+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:37.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:37.635+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:37.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:37.635+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:37.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:37.636+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:40:38.676 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:38.536+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:38.676 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:38.536+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:38.676 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:38.536+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:38.676 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:38.537+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:40:38.677 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:38.379+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:38.677 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:38.379+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:38.677 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:38.379+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:38.677 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:38.380+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:40:38.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:38.674+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:38.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:38.674+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:38.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:38.674+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:38.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:38.675+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:40:39.665 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:39.566+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:39.666 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:39.566+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:39.666 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:39.566+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:39.666 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:39.567+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:40:39.666 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:39.386+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:39.666 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:39.386+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:39.666 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:39.386+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:39.667 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:39.387+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:40:39.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:39.664+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:39.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:39.664+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:39.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:39.664+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:39.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:39.665+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:40:40.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:40.558+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:40.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:40.558+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:40.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:40.558+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:40.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:40.559+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:40:40.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:40.434+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:40.699 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:40.434+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:40.699 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:40.434+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:40.699 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:40.435+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:40:40.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:40.652+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:40.700 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:40.652+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:40.700 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:40.652+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:40.700 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:40.653+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:40:41.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:41.472+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:41.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:41.472+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:41.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:41.472+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:41.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:41.473+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:40:41.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:41.581+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:41.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:41.581+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:41.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:41.581+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:41.699 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:41.582+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:40:41.699 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:40:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:40:41.417+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 21822 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:40:42.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:41.701+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:42.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:41.701+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:42.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:41.701+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:42.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:41.702+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:40:42.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:42.585+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:42.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:42.585+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:42.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:42.585+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:42.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:42.586+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:40:42.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:42.488+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:42.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:42.488+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:42.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:42.488+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:42.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:42.488+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:40:42.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:42.663+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:42.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:42.663+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:42.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:42.663+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:42.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:42.663+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:40:43.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:43.696+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:43.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:43.696+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:43.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:43.696+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:43.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:43.697+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:40:43.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:43.545+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:43.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:43.545+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:43.949 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:43.545+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:43.949 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:43.546+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:40:43.949 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:43.467+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:43.949 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:43.467+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:43.949 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:43.467+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:43.949 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:43.468+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:40:44.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:44.503+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:44.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:44.503+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:44.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:44.503+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:44.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:44.503+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:40:44.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:44.423+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:44.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:44.423+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:44.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:44.423+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:44.699 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:44.424+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:40:44.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:44.651+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:44.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:44.651+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:44.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:44.651+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:44.699 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:44.652+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:40:45.661 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:45.486+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:45.661 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:45.486+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:45.661 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:45.486+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:45.661 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:45.487+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:40:45.661 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:45.401+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:45.662 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:45.401+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:45.662 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:45.401+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:45.662 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:45.402+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:40:45.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:45.659+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:45.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:45.659+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:45.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:45.659+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:45.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:45.660+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:40:46.622 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:40:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:40:46.420+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 21914 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:40:46.623 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:46.619+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:46.623 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:46.619+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:46.623 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:46.619+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:46.623 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:46.620+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:40:46.623 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:46.470+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:46.623 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:46.470+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:46.624 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:46.470+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:46.624 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:46.471+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:40:46.624 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:46.370+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:46.624 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:46.370+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:46.624 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:46.370+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:46.624 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:46.371+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:40:47.629 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:47.356+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:47.630 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:47.356+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:47.630 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:47.356+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:47.630 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:47.357+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:40:47.630 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:47.424+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:47.630 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:47.424+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:47.630 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:47.424+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:47.630 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:47.425+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:40:47.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:47.628+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:47.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:47.628+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:47.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:47.628+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:47.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:47.629+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:40:48.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:48.324+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:48.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:48.324+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:48.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:48.324+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:48.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:48.325+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:40:48.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:48.456+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:48.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:48.456+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:48.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:48.456+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:48.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:48.457+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:40:48.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:48.652+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:48.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:48.652+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:48.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:48.652+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:48.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:48.653+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:40:49.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:49.280+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:49.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:49.280+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:49.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:49.280+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:49.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:49.281+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:40:49.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:49.453+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:49.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:49.454+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:49.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:49.454+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:49.699 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:49.455+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:40:50.177 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:49.702+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:50.178 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:49.702+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:50.178 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:49.702+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:50.178 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:49.703+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:40:50.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:50.232+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:50.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:50.232+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:50.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:50.232+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:50.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:50.233+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:40:50.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:50.497+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:50.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:50.497+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:50.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:50.497+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:50.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:50.498+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:40:50.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:50.721+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:50.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:50.721+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:50.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:50.721+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:50.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:50.722+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:40:51.482 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:40:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:40:51.422+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 22003 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:40:51.483 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:51.212+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:51.483 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:51.212+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:51.483 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:51.212+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:51.483 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:51.213+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:40:51.760 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:51.481+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:51.760 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:51.481+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:51.760 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:51.481+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:51.761 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:51.482+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:40:52.188 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:51.758+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:52.189 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:51.758+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:52.189 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:51.758+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:52.189 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:51.759+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:40:52.450 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:52.187+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:52.450 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:52.187+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:52.450 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:52.187+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:52.450 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:52.188+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:40:52.797 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:52.486+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:52.797 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:52.486+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:52.797 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:52.486+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:52.797 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:52.487+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:40:53.157 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:52.795+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:53.157 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:52.795+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:53.157 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:52.795+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:53.157 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:52.796+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:40:53.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:53.217+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:53.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:53.217+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:53.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:53.217+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:53.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:53.218+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:40:53.797 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:53.446+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:53.798 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:53.446+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:53.798 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:53.446+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:53.798 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:53.447+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:40:54.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:53.796+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:54.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:53.796+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:54.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:53.796+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:54.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:53.797+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:40:54.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:54.253+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:54.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:54.253+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:54.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:54.253+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:54.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:54.254+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:40:54.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:54.454+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:54.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:54.454+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:54.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:54.454+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:54.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:54.455+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:40:55.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:54.754+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:55.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:54.754+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:55.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:54.754+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:55.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:54.754+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:40:55.505 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:55.224+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:55.505 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:55.224+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:55.506 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:55.224+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:55.506 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:55.225+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:40:55.806 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:55.504+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:55.806 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:55.504+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:55.806 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:55.504+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:55.806 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:55.505+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:40:56.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:55.802+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:56.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:55.802+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:56.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:55.802+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:56.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:55.803+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:40:56.484 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:40:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:40:56.425+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 22096 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:40:56.485 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:56.216+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:56.485 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:56.216+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:56.485 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:56.216+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:56.485 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:56.217+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:40:56.832 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:56.483+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:56.832 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:56.483+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:56.832 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:56.483+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:56.832 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:56.484+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:40:57.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:56.830+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:57.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:56.830+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:57.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:56.830+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:57.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:56.831+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:40:57.490 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:57.219+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:57.491 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:57.219+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:57.491 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:57.219+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:57.491 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:57.220+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:40:57.842 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:57.489+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:57.842 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:57.489+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:57.842 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:57.489+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:57.842 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:57.490+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:40:58.176 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:57.840+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:58.177 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:57.840+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:58.177 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:57.840+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:58.177 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:57.841+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:40:58.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:58.175+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:58.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:58.175+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:58.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:58.175+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:58.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:58.176+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:40:58.800 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:58.475+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:58.800 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:58.475+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:58.800 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:58.475+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:58.800 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:58.476+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:40:59.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:58.798+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:59.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:58.798+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:59.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:58.798+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:40:59.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:58.799+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:40:59.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:59.212+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:59.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:59.212+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:59.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:59.212+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:40:59.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:40:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:40:59.213+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:40:59.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:59.431+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:59.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:59.431+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:59.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:59.431+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:40:59.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:40:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:40:59.432+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:41:00.178 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:59.840+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:00.178 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:59.840+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:00.178 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:59.840+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:00.178 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:40:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:40:59.842+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:41:00.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:00.422+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:00.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:00.423+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:00.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:00.423+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:00.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:00.424+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:41:00.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:00.211+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:00.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:00.211+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:00.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:00.211+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:00.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:00.212+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:41:01.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:00.817+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:01.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:00.817+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:01.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:00.817+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:01.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:00.818+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:41:01.553 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:41:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:41:01.427+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 22185 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:41:01.554 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:01.441+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:01.554 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:01.441+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:01.554 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:01.441+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:01.554 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:01.442+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:41:01.554 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:01.245+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:01.555 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:01.245+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:01.555 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:01.245+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:01.555 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:01.246+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:41:02.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:01.828+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:02.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:01.828+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:02.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:01.828+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:02.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:01.830+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:41:02.587 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:02.394+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:02.588 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:02.394+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:02.588 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:02.394+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:02.588 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:02.395+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:41:02.588 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:02.277+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:02.588 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:02.277+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:02.589 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:02.277+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:02.589 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:02.278+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:41:02.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:02.806+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:02.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:02.806+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:02.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:02.806+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:02.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:02.807+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:41:03.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:03.291+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:03.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:03.291+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:03.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:03.291+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:03.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:03.292+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:41:03.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:03.355+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:03.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:03.355+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:03.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:03.355+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:03.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:03.356+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:41:04.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:03.854+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:04.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:03.854+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:04.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:03.854+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:04.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:03.855+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:41:04.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:04.272+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:04.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:04.272+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:04.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:04.272+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:04.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:04.272+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:41:04.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:04.383+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:04.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:04.383+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:04.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:04.383+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:04.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:04.384+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:41:05.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:04.851+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:05.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:04.851+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:05.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:04.851+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:05.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:04.852+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:41:05.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:05.306+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:05.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:05.306+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:05.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:05.306+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:05.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:05.307+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:41:05.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:05.372+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:05.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:05.372+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:05.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:05.372+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:05.699 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:05.373+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:41:06.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:05.845+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:06.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:05.845+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:06.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:05.845+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:06.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:05.845+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:41:06.697 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:41:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:41:06.430+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 22277 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:41:06.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:06.322+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:06.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:06.322+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:06.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:06.322+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:06.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:06.323+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:41:06.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:06.354+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:06.699 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:06.354+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:06.699 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:06.354+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:06.699 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:06.354+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:41:07.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:06.822+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:07.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:06.822+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:07.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:06.822+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:07.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:06.823+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:41:07.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:07.274+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:07.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:07.274+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:07.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:07.274+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:07.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:07.275+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:41:07.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:07.323+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:07.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:07.323+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:07.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:07.323+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:07.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:07.324+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:41:08.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:07.810+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:08.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:07.810+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:08.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:07.810+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:08.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:07.810+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:41:08.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:08.334+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:08.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:08.334+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:08.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:08.334+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:08.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:08.335+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:41:08.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:08.255+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:08.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:08.255+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:08.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:08.255+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:08.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:08.256+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:41:09.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:08.768+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:09.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:08.768+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:09.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:08.768+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:09.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:08.769+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:41:09.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:09.353+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:09.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:09.353+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:09.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:09.353+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:09.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:09.354+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:41:09.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:09.274+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:09.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:09.274+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:09.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:09.274+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:09.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:09.275+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:41:10.178 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:09.797+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:10.178 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:09.797+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:10.178 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:09.797+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:10.178 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:09.798+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:41:10.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:10.313+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:10.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:10.313+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:10.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:10.313+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:10.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:10.314+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:41:10.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:10.251+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:10.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:10.251+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:10.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:10.251+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:10.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:10.252+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:41:11.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:10.806+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:11.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:10.806+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:11.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:10.806+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:11.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:10.806+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:41:11.558 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:41:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:41:11.432+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 22366 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:41:11.559 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:11.231+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:11.559 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:11.231+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:11.559 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:11.231+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:11.559 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:11.232+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:41:11.560 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:11.333+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:11.560 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:11.333+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:11.560 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:11.333+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:11.560 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:11.334+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:41:11.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:11.785+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:11.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:11.785+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:11.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:11.785+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:11.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:11.786+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:41:12.540 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:12.324+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:12.540 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:12.324+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:12.540 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:12.324+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:12.540 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:12.325+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:41:12.540 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:12.218+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:12.541 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:12.218+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:12.541 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:12.218+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:12.541 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:12.218+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:41:12.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:12.779+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:12.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:12.779+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:12.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:12.779+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:12.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:12.780+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:41:13.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:13.343+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:13.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:13.343+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:13.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:13.343+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:13.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:13.344+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:41:13.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:13.223+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:13.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:13.223+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:13.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:13.223+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:13.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:13.224+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:41:14.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:13.760+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:14.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:13.760+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:14.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:13.760+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:14.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:13.760+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:41:14.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:14.309+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:14.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:14.309+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:14.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:14.309+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:14.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:14.310+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:41:14.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:14.236+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:14.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:14.236+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:14.699 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:14.236+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:14.699 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:14.237+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:41:15.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:14.788+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:15.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:14.788+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:15.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:14.788+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:15.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:14.789+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:41:15.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:15.319+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:15.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:15.319+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:15.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:15.319+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:15.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:15.320+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:41:15.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:15.261+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:15.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:15.261+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:15.699 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:15.261+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:15.699 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:15.262+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:41:16.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:15.829+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:16.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:15.829+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:16.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:15.829+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:16.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:15.830+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:41:16.560 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:41:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:41:16.435+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 22459 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:41:16.561 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:16.328+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:16.561 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:16.328+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:16.561 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:16.328+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:16.562 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:16.329+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:41:16.562 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:16.285+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:16.562 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:16.285+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:16.562 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:16.285+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:16.562 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:16.285+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:41:17.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:16.874+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:17.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:16.874+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:17.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:16.874+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:17.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:16.875+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:41:17.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:17.249+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:17.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:17.249+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:17.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:17.249+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:17.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:17.250+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:41:17.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:17.326+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:17.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:17.326+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:17.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:17.326+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:17.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:17.327+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:41:18.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:17.878+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:18.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:17.878+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:18.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:17.878+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:18.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:17.879+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:41:18.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:18.206+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:18.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:18.206+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:18.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:18.206+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:18.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:18.206+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:41:18.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:18.278+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:18.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:18.278+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:18.699 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:18.278+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:18.699 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:18.278+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:41:19.179 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:18.844+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:19.179 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:18.844+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:19.179 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:18.844+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:19.179 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:18.844+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:41:19.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:19.307+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:19.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:19.307+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:19.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:19.307+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:19.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:19.307+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:41:19.449 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:19.177+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:19.449 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:19.177+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:19.450 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:19.177+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:19.450 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:19.178+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:41:20.178 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:19.820+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:20.179 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:19.820+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:20.179 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:19.820+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:20.179 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:19.821+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:41:20.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:20.300+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:20.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:20.300+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:20.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:20.300+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:20.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:20.301+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:41:20.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:20.177+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:20.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:20.177+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:20.449 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:20.177+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:20.449 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:20.178+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:41:21.149 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:20.802+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:21.149 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:20.802+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:21.149 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:20.802+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:21.150 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:20.803+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:41:21.439 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:21.317+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:21.439 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:21.317+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:21.439 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:21.317+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:21.439 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:21.318+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:41:21.439 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:21.147+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:21.440 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:21.147+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:21.440 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:21.147+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:21.440 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:21.148+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:41:21.697 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:41:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:41:21.437+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 22547 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:41:22.162 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:21.782+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:22.162 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:21.783+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:22.162 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:21.783+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:22.162 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:21.783+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:41:22.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:22.315+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:22.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:22.315+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:22.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:22.315+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:22.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:22.316+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:41:22.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:22.160+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:22.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:22.160+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:22.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:22.160+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:22.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:22.161+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:41:23.185 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:22.813+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:23.186 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:22.813+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:23.186 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:22.813+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:23.186 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:22.813+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:41:23.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:23.184+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:23.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:23.184+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:23.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:23.184+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:23.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:23.185+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:41:23.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:23.321+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:23.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:23.321+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:23.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:23.321+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:23.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:23.321+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:41:24.152 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:23.800+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:24.152 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:23.800+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:24.153 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:23.800+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:24.153 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:23.801+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:41:24.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:24.356+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:24.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:24.356+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:24.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:24.356+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:24.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:24.357+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:41:24.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:24.150+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:24.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:24.150+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:24.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:24.150+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:24.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:24.151+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:41:25.127 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:24.782+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:25.127 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:24.782+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:25.127 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:24.782+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:25.127 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:24.784+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:41:25.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:25.356+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:25.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:25.356+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:25.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:25.356+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:25.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:25.357+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:41:25.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:25.125+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:25.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:25.125+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:25.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:25.125+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:25.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:25.126+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:41:26.153 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:25.793+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:26.154 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:25.793+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:26.154 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:25.793+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:26.154 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:25.793+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:41:26.441 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:26.353+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:26.441 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:26.353+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:26.441 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:26.353+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:26.442 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:26.353+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:41:26.442 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:26.152+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:26.442 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:26.152+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:26.442 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:26.152+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:26.442 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:26.153+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:41:26.697 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:41:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:41:26.439+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 22639 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:41:27.174 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:26.777+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:27.174 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:26.777+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:27.174 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:26.777+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:27.174 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:26.778+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:41:27.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:27.345+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:27.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:27.345+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:27.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:27.345+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:27.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:27.346+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:41:27.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:27.172+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:27.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:27.172+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:27.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:27.172+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:27.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:27.173+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:41:28.147 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:27.818+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:28.147 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:27.818+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:28.147 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:27.818+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:28.147 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:27.819+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:41:28.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:28.145+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:28.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:28.145+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:28.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:28.145+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:28.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:28.146+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:41:28.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:28.359+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:28.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:28.359+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:28.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:28.359+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:28.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:28.360+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:41:29.150 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:28.784+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:29.151 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:28.784+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:29.151 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:28.784+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:29.151 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:28.785+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:41:29.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:29.350+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:29.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:29.350+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:29.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:29.350+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:29.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:29.351+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:41:29.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:29.149+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:29.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:29.149+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:29.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:29.149+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:29.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:29.150+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:41:30.160 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:29.744+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:30.160 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:29.744+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:30.160 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:29.744+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:30.161 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:29.744+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:41:30.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:30.159+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:30.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:30.159+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:30.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:30.159+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:30.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:30.159+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:41:30.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:30.341+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:30.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:30.341+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:30.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:30.341+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:30.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:30.341+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:41:31.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:30.780+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:31.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:30.780+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:31.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:30.780+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:31.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:30.780+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:41:31.465 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:41:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:41:31.442+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 22728 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:41:31.465 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:31.205+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:31.465 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:31.205+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:31.465 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:31.205+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:31.466 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:31.206+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:41:31.466 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:31.374+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:31.466 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:31.374+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:31.466 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:31.374+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:31.466 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:31.375+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:41:32.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:31.779+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:32.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:31.779+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:32.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:31.779+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:32.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:31.780+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:41:32.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:32.203+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:32.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:32.203+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:32.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:32.203+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:32.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:32.204+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:41:32.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:32.402+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:32.699 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:32.402+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:32.699 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:32.402+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:32.699 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:32.403+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:41:33.166 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:32.788+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:33.166 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:32.788+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:33.166 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:32.788+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:33.166 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:32.789+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:41:33.438 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:33.164+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:33.438 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:33.164+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:33.439 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:33.164+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:33.439 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:33.165+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:41:33.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:33.437+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:33.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:33.437+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:33.699 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:33.437+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:33.699 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:33.437+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:41:34.171 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:33.743+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:34.171 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:33.743+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:34.172 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:33.743+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:34.172 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:33.744+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:41:34.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:34.170+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:34.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:34.170+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:34.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:34.170+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:34.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:34.171+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:41:34.791 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:34.451+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:34.791 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:34.451+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:34.792 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:34.451+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:34.792 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:34.452+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:41:35.151 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:34.789+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:35.152 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:34.789+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:35.152 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:34.789+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:35.152 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:34.790+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:41:35.437 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:35.150+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:35.437 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:35.150+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:35.438 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:35.150+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:35.438 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:35.151+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:41:35.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:35.436+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:35.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:35.436+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:35.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:35.436+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:35.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:35.437+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:41:36.194 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:35.773+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:36.195 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:35.773+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:36.195 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:35.773+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:36.195 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:35.774+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:41:36.446 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:36.193+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:36.446 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:36.193+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:36.446 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:36.193+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:36.446 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:36.194+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:41:36.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:36.415+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:36.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:36.415+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:36.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:36.415+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:36.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:36.415+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:41:36.697 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:41:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:41:36.444+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 22821 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:41:37.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:36.745+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:37.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:36.745+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:37.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:36.745+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:37.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:36.746+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:41:37.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:37.446+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:37.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:37.446+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:37.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:37.446+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:37.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:37.447+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:41:37.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:37.214+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:37.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:37.214+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:37.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:37.214+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:37.699 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:37.215+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:41:38.168 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:37.758+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:38.168 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:37.758+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:38.168 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:37.758+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:38.169 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:37.759+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:41:38.419 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:38.167+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:38.419 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:38.167+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:38.420 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:38.167+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:38.420 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:38.168+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:41:38.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:38.418+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:38.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:38.418+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:38.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:38.418+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:38.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:38.418+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:41:39.180 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:38.795+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:39.181 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:38.795+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:39.181 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:38.795+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:39.181 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:38.795+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:41:39.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:39.422+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:39.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:39.422+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:39.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:39.422+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:39.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:39.423+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:41:39.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:39.179+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:39.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:39.179+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:39.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:39.179+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:39.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:39.180+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:41:40.174 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:39.812+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:40.174 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:39.812+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:40.174 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:39.812+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:40.174 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:39.813+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:41:40.431 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:40.172+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:40.431 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:40.172+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:40.432 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:40.172+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:40.432 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:40.173+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:41:40.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:40.429+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:40.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:40.429+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:40.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:40.429+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:40.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:40.430+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:41:41.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:40.780+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:41.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:40.780+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:41.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:40.780+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:41.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:40.781+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:41:41.461 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:41:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:41:41.447+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 22910 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:41:41.461 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:41.405+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:41.461 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:41.405+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:41.461 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:41.405+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:41.462 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:41.406+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:41:41.462 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:41.198+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:41.462 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:41.198+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:41.462 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:41.198+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:41.462 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:41.199+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:41:42.192 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:41.817+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:42.193 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:41.817+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:42.193 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:41.817+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:42.193 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:41.818+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:41:42.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:42.191+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:42.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:42.191+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:42.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:42.191+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:42.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:42.192+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:41:42.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:42.360+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:42.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:42.360+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:42.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:42.360+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:42.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:42.361+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:41:43.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:42.804+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:43.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:42.804+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:43.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:42.804+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:43.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:42.805+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:41:43.615 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:43.332+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:43.615 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:43.332+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:43.615 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:43.332+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:43.615 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:43.333+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:41:43.616 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:43.196+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:43.616 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:43.196+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:43.616 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:43.196+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:43.616 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:43.197+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:41:43.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:43.807+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:43.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:43.807+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:43.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:43.807+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:43.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:43.808+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:41:44.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:44.321+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:44.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:44.321+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:44.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:44.321+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:44.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:44.321+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:41:44.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:44.188+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:44.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:44.188+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:44.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:44.188+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:44.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:44.189+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:41:45.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:44.798+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:45.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:44.798+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:45.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:44.798+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:45.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:44.799+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:41:45.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:45.368+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:45.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:45.368+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:45.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:45.368+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:45.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:45.369+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:41:45.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:45.237+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:45.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:45.237+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:45.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:45.237+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:45.699 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:45.238+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:41:46.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:45.788+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:46.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:45.788+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:46.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:45.788+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:46.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:45.789+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:41:46.434 ERROR:paramiko.transport:Socket exception: No route to host (113) 2024-05-04T12:41:46.439 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-04T12:41:46.439 INFO:tasks.workunit:Stopping ['suites/fsstress.sh'] on client.1... 2024-05-04T12:41:46.440 ERROR:teuthology.run_tasks:Manager failed: kclient Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_09dbd6bba52d19ba471c33bc15c008e6ad158ea6/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_ceph_ceph-c_09dbd6bba52d19ba471c33bc15c008e6ad158ea6/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_ceph_ceph-c_09dbd6bba52d19ba471c33bc15c008e6ad158ea6/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host 2024-05-04T12:41:46.453 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-05-04T12:41:46.464 INFO:tasks.cephadm:Teardown begin 2024-05-04T12:41:46.464 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_09dbd6bba52d19ba471c33bc15c008e6ad158ea6/qa/tasks/cephadm.py", line 2240, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_09dbd6bba52d19ba471c33bc15c008e6ad158ea6/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_ceph_ceph-c_09dbd6bba52d19ba471c33bc15c008e6ad158ea6/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_ceph_ceph-c_09dbd6bba52d19ba471c33bc15c008e6ad158ea6/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host 2024-05-04T12:41:46.469 DEBUG:teuthology.orchestra.run.smithi079:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-04T12:41:46.501 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-05-04T12:41:46.502 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-04T12:41:46.573 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:41:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:41:46.450+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 23001 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:41:46.573 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:46.349+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:46.573 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:46.349+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:46.574 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:46.349+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:46.574 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:46.349+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:41:46.574 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:46.265+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:46.574 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:46.265+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:46.574 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:46.265+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:46.574 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:46.266+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:41:46.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:46.749+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:46.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:46.749+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:46.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:46.749+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:46.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:46.750+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:41:47.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:47.265+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:47.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:47.265+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:47.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:47.265+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:47.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:47.266+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:41:47.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:47.341+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:47.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:47.341+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:47.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:47.341+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:47.699 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:47.342+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:41:48.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:47.729+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:48.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:47.729+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:48.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:47.729+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:48.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:47.730+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:41:48.658 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:48.333+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:48.658 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:48.333+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:48.659 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:48.333+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:48.659 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:48.334+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:41:48.659 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:48.240+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:48.659 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:48.240+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:48.659 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:48.240+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:48.659 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:48.240+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:41:48.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:48.757+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:48.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:48.757+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:48.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:48.757+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:48.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:48.758+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:41:49.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:49.297+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:49.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:49.297+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:49.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:49.297+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:49.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:49.298+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:41:49.449 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:49.195+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:49.449 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:49.195+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:49.449 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:49.195+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:49.449 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:49.196+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:41:49.570 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.146 2024-05-04T12:41:50.178 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:49.772+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:50.178 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:49.772+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:50.179 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:49.772+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:50.179 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:49.773+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:41:50.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:50.322+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:50.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:50.322+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:50.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:50.322+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:50.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:50.323+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:41:50.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:50.176+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:50.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:50.176+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:50.449 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:50.176+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:50.449 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:50.177+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:41:51.172 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:50.748+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:51.172 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:50.748+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:51.173 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:50.748+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:51.173 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:50.749+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:41:51.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:51.170+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:51.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:51.170+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:51.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:51.170+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:51.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:51.171+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:41:51.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:51.301+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:51.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:51.301+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:51.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:51.301+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:51.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:51.302+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:41:51.708 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:41:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:41:51.452+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 23090 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:41:52.177 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:51.706+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:52.178 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:51.706+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:52.178 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:51.706+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:52.178 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:51.707+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:41:52.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:52.267+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:52.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:52.267+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:52.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:52.267+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:52.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:52.268+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:41:52.449 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:52.176+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:52.449 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:52.176+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:52.449 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:52.176+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:52.449 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:52.177+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:41:52.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:52.659+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:52.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:52.659+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:52.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:52.659+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:52.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:52.660+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:41:53.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:53.204+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:53.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:53.204+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:53.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:53.204+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:53.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:53.205+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:41:53.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:53.253+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:53.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:53.253+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:53.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:53.253+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:53.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:53.254+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:41:54.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:53.704+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:54.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:53.704+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:54.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:53.704+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:54.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:53.705+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:41:54.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:54.200+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:54.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:54.200+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:54.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:54.200+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:54.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:54.201+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:41:54.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:54.248+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:54.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:54.248+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:54.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:54.248+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:54.699 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:54.249+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:41:55.156 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:54.712+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:55.157 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:54.712+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:55.157 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:54.712+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:55.157 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:54.713+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:41:55.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:55.155+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:55.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:55.155+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:55.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:55.155+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:55.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:55.156+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:41:55.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:55.244+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:55.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:55.244+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:55.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:55.244+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:55.449 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:55.245+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:41:56.184 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:55.744+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:56.184 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:55.744+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:56.184 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:55.744+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:56.184 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:55.745+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:41:56.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:56.213+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:56.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:56.213+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:56.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:56.213+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:56.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:56.214+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:41:56.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:56.182+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:56.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:56.182+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:56.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:56.182+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:56.449 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:56.183+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:41:56.773 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:41:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:41:56.454+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 23183 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:41:57.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:56.772+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:57.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:56.772+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:57.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:56.772+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:57.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:56.773+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:41:57.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:57.224+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:57.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:57.224+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:57.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:57.224+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:57.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:57.225+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:41:57.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:57.220+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:57.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:57.220+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:57.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:57.220+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:57.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:57.221+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:41:58.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:57.807+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:58.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:57.807+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:58.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:57.807+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:58.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:57.807+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:41:58.571 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-05-04T12:41:58.572 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-04T12:41:58.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:58.200+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:58.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:58.200+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:58.699 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:58.200+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:58.699 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:58.201+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:41:58.699 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:58.260+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:58.699 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:58.260+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:58.700 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:58.260+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:58.700 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:58.261+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:41:59.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:58.775+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:59.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:58.775+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:59.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:58.775+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:41:59.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:58.776+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:41:59.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:59.203+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:59.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:59.203+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:59.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:59.203+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:41:59.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:41:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:41:59.203+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:41:59.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:59.297+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:59.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:59.297+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:59.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:59.297+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:41:59.699 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:41:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:41:59.298+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:42:00.168 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:59.778+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:00.169 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:59.778+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:00.169 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:59.778+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:00.169 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:41:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:41:59.778+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:42:00.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:00.277+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:00.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:00.277+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:00.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:00.277+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:00.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:00.278+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:42:00.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:00.167+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:00.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:00.167+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:00.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:00.167+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:00.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:00.168+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:42:01.163 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:00.774+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:01.163 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:00.774+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:01.163 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:00.774+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:01.163 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:00.774+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:42:01.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:01.162+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:01.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:01.162+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:01.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:01.162+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:01.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:01.163+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:42:01.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:01.232+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:01.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:01.232+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:01.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:01.232+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:01.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:01.233+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:42:01.630 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.146 2024-05-04T12:42:01.801 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:42:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:42:01.458+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 23272 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:42:02.116 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:01.800+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:02.116 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:01.800+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:02.116 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:01.800+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:02.116 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:01.801+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:42:02.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:02.115+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:02.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:02.115+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:02.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:02.115+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:02.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:02.116+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:42:02.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:02.190+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:02.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:02.190+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:02.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:02.190+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:02.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:02.191+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:42:03.089 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:02.773+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:03.089 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:02.773+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:03.089 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:02.773+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:03.089 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:02.774+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:42:03.416 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:03.169+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:03.416 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:03.169+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:03.416 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:03.169+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:03.417 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:03.170+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:42:03.417 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:03.087+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:03.417 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:03.087+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:03.418 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:03.087+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:03.418 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:03.088+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:42:04.086 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:03.801+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:04.086 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:03.801+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:04.086 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:03.801+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:04.087 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:03.802+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:42:04.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:04.096+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:04.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:04.096+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:04.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:04.096+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:04.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:04.096+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:42:04.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:04.125+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:04.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:04.125+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:04.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:04.125+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:04.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:04.126+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:42:05.087 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:04.816+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:05.087 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:04.816+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:05.087 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:04.816+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:05.087 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:04.818+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:42:05.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:05.114+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:05.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:05.114+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:05.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:05.114+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:05.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:05.115+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:42:05.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:05.085+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:05.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:05.085+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:05.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:05.085+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:05.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:05.086+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:42:06.116 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:05.772+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:06.117 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:05.772+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:06.117 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:05.772+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:06.117 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:05.773+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:42:06.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:06.124+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:06.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:06.124+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:06.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:06.124+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:06.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:06.125+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:42:06.449 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:06.133+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:06.449 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:06.133+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:06.450 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:06.133+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:06.450 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:06.134+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:42:06.783 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:42:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:42:06.460+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 23364 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:42:07.098 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:06.781+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:07.099 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:06.781+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:07.099 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:06.781+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:07.099 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:06.782+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:42:07.416 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:07.097+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:07.416 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:07.097+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:07.417 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:07.097+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:07.417 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:07.098+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:42:07.417 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:07.109+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:07.417 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:07.109+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:07.417 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:07.109+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:07.417 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:07.110+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:42:08.107 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:07.796+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:08.107 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:07.796+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:08.107 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:07.796+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:08.107 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:07.798+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:42:08.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:08.105+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:08.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:08.105+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:08.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:08.105+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:08.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:08.106+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:42:08.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:08.146+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:08.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:08.146+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:08.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:08.146+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:08.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:08.147+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:42:09.146 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:08.826+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:09.146 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:08.826+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:09.147 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:08.826+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:09.147 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:08.827+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:42:09.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:09.146+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:09.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:09.146+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:09.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:09.146+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:09.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:09.147+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:42:09.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:09.144+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:09.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:09.144+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:09.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:09.144+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:09.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:09.145+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:42:10.123 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:09.855+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:10.123 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:09.855+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:10.123 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:09.855+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:10.123 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:09.856+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:42:10.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:10.148+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:10.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:10.148+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:10.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:10.148+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:10.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:10.148+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:42:10.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:10.121+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:10.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:10.121+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:10.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:10.121+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:10.449 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:10.122+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:42:11.158 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:10.879+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:11.159 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:10.879+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:11.159 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:10.879+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:11.159 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:10.880+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:42:11.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:11.158+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:11.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:11.158+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:11.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:11.158+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:11.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:11.158+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:42:11.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:11.157+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:11.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:11.157+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:11.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:11.157+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:11.449 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:11.158+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:42:11.886 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:42:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:42:11.462+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 23453 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:42:12.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:11.884+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:12.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:11.884+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:12.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:11.884+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:12.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:11.885+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:42:12.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:12.131+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:12.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:12.131+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:12.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:12.131+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:12.199 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:12.132+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:42:12.199 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:12.115+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:12.199 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:12.115+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:12.199 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:12.115+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:12.199 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:12.116+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:42:13.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:13.117+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:13.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:13.117+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:13.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:13.117+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:13.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:13.118+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:42:13.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:13.139+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:13.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:13.139+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:13.199 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:13.139+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:13.199 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:13.140+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:42:13.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:12.900+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:13.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:12.900+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:13.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:12.900+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:13.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:12.901+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:42:13.631 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-05-04T12:42:13.632 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-04T12:42:14.135 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:13.878+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:14.135 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:13.878+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:14.135 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:13.878+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:14.136 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:13.879+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:42:14.136 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:14.090+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:14.136 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:14.090+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:14.136 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:14.090+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:14.136 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:14.091+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:42:14.435 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:14.133+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:14.435 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:14.133+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:14.436 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:14.133+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:14.436 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:14.134+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:42:15.073 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:14.837+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:15.073 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:14.837+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:15.074 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:14.837+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:15.074 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:14.838+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:42:15.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:15.154+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:15.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:15.154+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:15.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:15.154+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:15.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:15.154+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:42:15.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:15.072+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:15.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:15.072+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:15.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:15.072+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:15.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:15.073+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:42:16.116 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:15.827+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:16.116 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:15.827+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:16.116 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:15.827+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:16.116 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:15.828+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:42:16.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:16.114+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:16.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:16.114+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:16.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:16.114+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:16.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:16.115+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:42:16.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:16.200+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:16.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:16.200+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:16.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:16.201+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:16.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:16.201+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:42:16.702 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.146 2024-05-04T12:42:16.810 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:42:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:42:16.465+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 23546 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:42:17.163 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:16.823+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:17.163 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:16.823+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:17.163 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:16.823+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:17.164 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:16.824+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:42:17.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:17.217+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:17.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:17.217+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:17.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:17.217+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:17.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:17.218+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:42:17.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:17.161+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:17.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:17.161+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:17.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:17.161+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:17.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:17.162+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:42:18.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:17.843+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:18.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:17.843+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:18.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:17.843+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:18.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:17.844+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:42:18.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:18.225+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:18.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:18.225+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:18.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:18.225+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:18.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:18.226+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:42:18.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:18.210+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:18.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:18.210+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:18.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:18.210+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:18.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:18.211+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:42:19.183 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:18.814+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:19.183 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:18.814+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:19.183 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:18.814+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:19.183 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:18.816+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:42:19.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:19.199+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:19.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:19.199+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:19.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:19.199+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:19.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:19.200+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:42:19.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:19.181+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:19.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:19.181+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:19.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:19.181+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:19.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:19.182+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:42:19.703 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-05-04T12:42:19.704 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-04T12:42:19.778 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.146 2024-05-04T12:42:19.778 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-05-04T12:42:19.779 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-05-04T12:42:19.779 DEBUG:teuthology.orchestra.run.smithi079:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-05-04T12:42:19.796 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-05-04T12:42:19.796 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-04T12:42:20.162 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:19.803+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:20.162 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:19.803+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:20.162 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:19.803+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:20.162 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:19.804+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:42:20.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:20.177+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:20.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:20.177+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:20.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:20.177+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:20.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:20.178+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:42:20.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:20.160+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:20.449 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:20.160+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:20.449 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:20.160+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:20.449 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:20.161+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:42:21.187 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:20.816+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:21.187 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:20.816+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:21.187 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:20.816+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:21.188 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:20.816+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:42:21.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:21.215+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:21.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:21.215+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:21.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:21.215+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:21.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:21.216+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:42:21.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:21.186+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:21.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:21.186+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:21.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:21.186+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:21.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:21.186+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:42:21.848 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:42:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:42:21.468+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 23635 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:42:22.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:21.846+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:22.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:21.846+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:22.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:21.846+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:22.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:21.847+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:42:22.460 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:22.235+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:22.460 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:22.235+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:22.460 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:22.235+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:22.460 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:22.236+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:42:22.460 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:22.199+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:22.461 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:22.199+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:22.461 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:22.199+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:22.461 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:22.200+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:42:22.846 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.146 2024-05-04T12:42:23.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:22.836+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:23.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:22.836+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:23.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:22.836+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:23.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:22.837+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:42:23.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:23.196+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:23.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:23.196+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:23.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:23.196+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:23.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:23.197+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:42:23.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:23.199+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:23.699 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:23.199+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:23.699 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:23.199+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:23.699 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:23.199+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:42:24.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:23.824+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:24.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:23.824+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:24.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:23.824+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:24.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:23.825+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:42:24.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:24.195+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:24.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:24.195+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:24.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:24.195+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:24.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:24.196+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:42:24.699 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:24.231+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:24.699 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:24.231+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:24.700 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:24.231+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:24.700 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:24.232+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:42:25.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:24.837+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:25.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:24.837+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:25.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:24.837+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:25.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:24.838+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:42:25.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:25.228+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:25.697 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:25.228+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:25.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:25.228+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:25.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:25.228+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:42:25.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:25.239+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:25.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:25.239+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:25.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:25.239+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:25.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:25.239+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:42:26.183 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:25.857+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:26.183 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:25.857+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:26.184 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:25.857+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:26.184 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:25.859+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:42:26.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:26.182+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:26.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:26.182+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:26.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:26.182+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:26.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:26.183+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:42:26.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:26.266+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:26.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:26.266+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:26.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:26.266+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:26.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:26.267+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:42:26.837 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:42:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:42:26.470+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 23726 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:42:27.180 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:26.835+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:27.181 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:26.835+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:27.181 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:26.835+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:27.181 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:26.836+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:42:27.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:27.256+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:27.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:27.256+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:27.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:27.256+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:27.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:27.257+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:42:27.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:27.179+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:27.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:27.179+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:27.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:27.179+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:27.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:27.180+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:42:28.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:27.845+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:28.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:27.845+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:28.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:27.845+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:28.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:27.845+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:42:28.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:28.251+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:28.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:28.251+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:28.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:28.251+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:28.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:28.252+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:42:28.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:28.224+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:28.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:28.224+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:28.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:28.224+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:28.699 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:28.224+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:42:29.186 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:28.857+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:29.186 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:28.857+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:29.186 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:28.857+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:29.186 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:28.857+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:42:29.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:29.183+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:29.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:29.183+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:29.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:29.183+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:29.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:29.184+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:42:29.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:29.284+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:29.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:29.284+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:29.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:29.284+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:29.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:29.285+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:42:30.159 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:29.830+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:30.159 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:29.830+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:30.160 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:29.830+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:30.160 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:29.831+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:42:30.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:30.158+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:30.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:30.158+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:30.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:30.158+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:30.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:30.159+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:42:30.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:30.275+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:30.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:30.275+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:30.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:30.275+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:30.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:30.276+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:42:31.166 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:30.802+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:31.166 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:30.802+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:31.166 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:30.802+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:31.166 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:30.803+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:42:31.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:31.272+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:31.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:31.272+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:31.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:31.272+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:31.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:31.272+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:42:31.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:31.164+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:31.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:31.164+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:31.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:31.164+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:31.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:31.166+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:42:31.826 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:42:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:42:31.473+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 23815 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:42:31.848 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-05-04T12:42:31.849 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-04T12:42:32.142 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:31.825+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:32.143 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:31.825+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:32.143 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:31.825+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:32.143 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:31.826+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:42:32.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:32.295+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:32.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:32.295+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:32.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:32.295+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:32.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:32.296+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:42:32.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:32.141+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:32.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:32.141+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:32.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:32.141+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:32.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:32.142+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:42:33.175 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:32.785+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:33.176 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:32.785+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:33.176 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:32.785+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:33.176 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:32.785+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:42:33.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:33.314+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:33.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:33.314+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:33.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:33.314+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:33.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:33.315+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:42:33.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:33.174+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:33.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:33.174+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:33.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:33.174+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:33.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:33.175+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:42:34.153 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:33.811+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:34.153 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:33.811+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:34.153 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:33.811+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:34.154 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:33.811+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:42:34.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:34.301+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:34.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:34.301+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:34.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:34.301+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:34.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:34.302+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:42:34.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:34.151+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:34.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:34.151+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:34.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:34.151+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:34.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:34.152+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:42:34.915 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.146 2024-05-04T12:42:35.180 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:34.842+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:35.180 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:34.842+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:35.181 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:34.842+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:35.181 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:34.842+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:42:35.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:35.307+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:35.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:35.307+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:35.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:35.307+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:35.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:35.307+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:42:35.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:35.178+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:35.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:35.178+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:35.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:35.178+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:35.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:35.179+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:42:36.187 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:35.811+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:36.188 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:35.811+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:36.188 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:35.811+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:36.188 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:35.812+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:42:36.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:36.286+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:36.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:36.286+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:36.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:36.286+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:36.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:36.287+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:42:36.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:36.186+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:36.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:36.186+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:36.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:36.186+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:36.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:36.187+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:42:36.851 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:42:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:42:36.475+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 23908 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:42:37.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:36.850+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:37.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:36.850+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:37.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:36.850+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:37.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:36.851+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:42:37.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:37.296+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:37.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:37.296+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:37.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:37.296+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:37.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:37.297+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:42:37.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:37.206+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:37.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:37.206+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:37.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:37.206+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:37.699 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:37.207+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:42:38.182 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:37.841+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:38.182 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:37.841+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:38.183 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:37.841+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:38.183 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:37.842+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:42:38.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:38.181+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:38.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:38.181+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:38.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:38.181+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:38.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:38.182+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:42:38.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:38.308+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:38.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:38.308+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:38.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:38.308+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:38.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:38.309+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:42:39.187 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:38.876+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:39.187 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:38.876+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:39.188 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:38.876+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:39.188 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:38.877+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:42:39.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:39.306+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:39.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:39.306+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:39.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:39.306+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:39.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:39.307+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:42:39.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:39.185+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:39.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:39.185+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:39.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:39.185+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:39.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:39.187+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:42:40.185 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:39.829+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:40.186 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:39.829+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:40.186 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:39.829+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:40.186 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:39.830+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:42:40.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:40.232+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:40.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:40.232+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:40.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:40.232+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:40.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:40.233+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:42:40.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:40.278+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:40.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:40.278+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:40.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:40.278+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:40.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:40.279+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:42:41.188 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:40.856+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:41.188 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:40.856+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:41.188 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:40.856+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:41.188 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:40.857+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:42:41.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:41.328+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:41.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:41.328+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:41.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:41.328+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:41.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:41.329+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:42:41.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:41.186+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:41.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:41.186+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:41.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:41.186+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:41.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:41.187+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:42:41.823 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:42:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:42:41.478+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 23997 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:42:42.154 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:41.822+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:42.154 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:41.822+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:42.154 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:41.822+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:42.154 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:41.823+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:42:42.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:42.286+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:42.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:42.286+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:42.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:42.286+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:42.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:42.287+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:42:42.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:42.152+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:42.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:42.152+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:42.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:42.152+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:42.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:42.153+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:42:43.151 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:42.784+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:43.152 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:42.784+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:43.152 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:42.784+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:43.152 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:42.785+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:42:43.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:43.314+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:43.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:43.314+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:43.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:43.314+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:43.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:43.315+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:42:43.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:43.150+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:43.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:43.150+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:43.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:43.150+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:43.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:43.151+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:42:44.142 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:43.765+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:44.142 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:43.765+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:44.143 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:43.765+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:44.143 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:43.766+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:42:44.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:44.316+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:44.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:44.316+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:44.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:44.316+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:44.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:44.317+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:42:44.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:44.144+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:44.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:44.144+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:44.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:44.144+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:44.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:44.145+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:42:45.102 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:44.719+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:45.103 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:44.719+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:45.103 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:44.719+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:45.103 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:44.719+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:42:45.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:45.350+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:45.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:45.350+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:45.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:45.350+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:45.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:45.350+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:42:45.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:45.101+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:45.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:45.101+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:45.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:45.101+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:45.449 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:45.102+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:42:46.084 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:45.720+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:46.085 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:45.720+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:46.085 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:45.720+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:46.085 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:45.720+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:42:46.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:46.318+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:46.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:46.318+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:46.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:46.318+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:46.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:46.319+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:42:46.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:46.084+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:46.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:46.084+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:46.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:46.084+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:46.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:46.085+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:42:46.916 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-05-04T12:42:46.917 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-04T12:42:46.947 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:42:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:42:46.481+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 24089 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:42:46.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:46.712+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:46.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:46.712+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:46.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:46.712+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:46.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:46.713+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:42:47.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:47.277+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:47.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:47.277+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:47.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:47.277+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:47.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:47.278+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:42:47.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:47.087+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:47.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:47.087+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:47.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:47.087+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:47.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:47.088+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:42:48.103 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:47.718+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:48.103 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:47.718+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:48.104 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:47.718+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:48.104 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:47.719+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:42:48.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:48.258+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:48.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:48.258+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:48.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:48.258+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:48.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:48.259+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:42:48.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:48.102+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:48.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:48.102+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:48.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:48.102+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:48.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:48.103+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:42:49.099 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:48.707+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:49.099 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:48.707+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:49.099 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:48.707+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:49.099 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:48.707+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:42:49.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:49.098+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:49.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:49.098+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:49.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:49.098+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:49.448 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:49.099+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:42:49.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:49.274+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:49.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:49.274+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:49.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:49.274+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:49.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:49.275+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:42:49.986 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.146 2024-05-04T12:42:50.070 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:49.723+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:50.070 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:49.723+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:50.071 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:49.723+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:50.071 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:49.724+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:42:50.321 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:50.318+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:50.321 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:50.318+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:50.322 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:50.318+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:50.322 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:50.320+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:42:50.322 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:50.068+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:50.322 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:50.068+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:50.322 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:50.069+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:50.322 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:50.070+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:42:51.084 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:50.726+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:51.085 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:50.726+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:51.085 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:50.726+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:51.085 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:50.727+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:42:51.353 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:51.083+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:51.353 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:51.083+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:51.353 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:51.083+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:51.353 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:51.084+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:42:51.608 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:42:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:42:51.483+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 24187 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:42:51.609 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:51.351+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:51.609 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:51.351+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:51.609 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:51.351+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:51.609 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:51.352+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:42:51.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:51.768+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:51.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:51.768+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:51.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:51.768+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:51.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:51.769+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:42:52.352 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:52.053+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:52.352 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:52.053+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:52.352 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:52.053+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:52.352 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:52.054+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:42:52.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:52.350+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:52.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:52.350+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:52.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:52.350+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:52.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:52.351+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:42:52.988 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-05-04T12:42:52.988 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-04T12:42:53.050 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:52.720+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:53.051 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:52.720+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:53.051 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:52.720+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:53.051 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:52.721+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:42:53.054 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.146 2024-05-04T12:42:53.054 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-05-04T12:42:53.055 INFO:tasks.cephadm:Archiving crash dumps... 2024-05-04T12:42:53.056 DEBUG:teuthology.misc:Transferring archived files from smithi079:/var/lib/ceph/d9b985c4-0a0e-11ef-bc97-c7b262605968/crash to /home/teuthworker/archive/yuriw-2024-05-02_19:10:09-fs-wip-yuri4-testing-2024-04-29-0642-distro-default-smithi/7686275/remote/smithi079/crash 2024-05-04T12:42:53.057 DEBUG:teuthology.orchestra.run.smithi079:> sudo tar c -f - -C /var/lib/ceph/d9b985c4-0a0e-11ef-bc97-c7b262605968/crash -- . 2024-05-04T12:42:53.083 DEBUG:teuthology.misc:Transferring archived files from smithi146:/var/lib/ceph/d9b985c4-0a0e-11ef-bc97-c7b262605968/crash to /home/teuthworker/archive/yuriw-2024-05-02_19:10:09-fs-wip-yuri4-testing-2024-04-29-0642-distro-default-smithi/7686275/remote/smithi146/crash 2024-05-04T12:42:53.084 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-05-04T12:42:53.084 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-04T12:42:53.345 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:53.049+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:53.345 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:53.049+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:53.345 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:53.049+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:53.345 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:53.050+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:42:53.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:53.343+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:53.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:53.343+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:53.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:53.343+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:53.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:53.344+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:42:54.041 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:53.728+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:54.042 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:53.728+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:54.042 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:53.728+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:54.042 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:53.729+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:42:54.365 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:54.040+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:54.366 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:54.040+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:54.366 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:54.040+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:54.366 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:54.041+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:42:54.684 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:54.364+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:54.684 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:54.364+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:54.684 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:54.364+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:54.684 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:54.365+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:42:54.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:54.682+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:54.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:54.682+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:54.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:54.682+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:54.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:54.683+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:42:55.357 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:55.075+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:55.357 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:55.075+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:55.358 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:55.075+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:55.358 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:55.076+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:42:55.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:55.356+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:55.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:55.356+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:55.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:55.356+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:55.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:55.357+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:42:56.045 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:55.718+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:56.045 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:55.718+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:56.045 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:55.718+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:56.045 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:55.718+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:42:56.158 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.146 2024-05-04T12:42:56.398 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:56.043+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:56.398 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:56.043+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:56.398 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:56.043+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:56.398 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:56.044+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:42:56.697 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:42:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:42:56.486+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 24279 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:42:56.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:56.396+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:56.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:56.396+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:56.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:56.396+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:56.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:56.397+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:42:57.000 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:56.715+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:57.000 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:56.715+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:57.000 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:56.715+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:57.000 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:56.716+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:42:57.404 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:56.998+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:57.404 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:56.998+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:57.404 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:56.998+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:57.404 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:56.999+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:42:57.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:57.402+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:57.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:57.402+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:57.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:57.402+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:57.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:57.403+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:42:58.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:57.979+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:58.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:57.979+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:58.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:57.979+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:58.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:57.980+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:42:58.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:57.759+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:58.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:57.759+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:58.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:57.759+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:58.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:57.760+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:42:58.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:58.423+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:58.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:58.423+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:58.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:58.423+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:58.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:58.424+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:42:59.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:58.937+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:59.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:58.937+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:59.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:58.937+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:42:59.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:58.937+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:42:59.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:58.777+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:59.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:58.777+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:59.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:58.777+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:42:59.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:58.778+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:42:59.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:59.392+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:59.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:59.392+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:59.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:59.392+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:42:59.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:42:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:42:59.392+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:43:00.186 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:59.912+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:00.187 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:59.912+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:00.187 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:59.912+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:00.187 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:42:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:42:59.912+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:43:00.187 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:59.824+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:00.187 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:59.824+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:00.188 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:59.824+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:00.188 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:42:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:42:59.824+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:43:00.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:00.381+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:00.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:00.381+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:00.447 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:00.381+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:00.448 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:00.382+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:43:01.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:00.887+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:01.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:00.887+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:01.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:00.887+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:01.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:00.888+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:43:01.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:00.797+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:01.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:00.797+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:01.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:00.797+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:01.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:00.798+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:43:01.615 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:43:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:43:01.489+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 24366 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:43:01.615 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:01.336+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:01.616 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:01.336+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:01.616 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:01.336+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:01.616 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:01.337+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:43:01.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:01.803+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:01.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:01.803+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:01.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:01.803+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:01.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:01.803+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:43:01.949 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:01.853+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:01.949 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:01.853+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:01.949 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:01.853+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:01.949 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:01.854+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:43:02.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:02.387+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:02.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:02.387+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:02.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:02.387+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:02.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:02.387+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:43:03.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:02.776+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:03.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:02.776+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:03.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:02.776+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:03.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:02.777+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:43:03.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:02.880+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:03.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:02.880+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:03.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:02.880+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:03.199 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:02.881+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:43:03.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:03.424+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:03.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:03.424+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:03.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:03.424+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:03.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:03.425+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:43:04.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:03.834+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:04.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:03.834+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:04.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:03.834+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:04.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:03.835+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:43:04.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:03.793+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:04.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:03.793+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:04.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:03.793+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:04.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:03.793+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:43:04.792 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:04.464+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:04.792 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:04.464+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:04.792 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:04.464+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:04.793 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:04.464+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:43:05.159 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-05-04T12:43:05.160 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-04T12:43:05.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:04.790+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:05.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:04.790+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:05.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:04.790+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:05.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:04.791+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:43:05.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:04.835+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:05.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:04.835+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:05.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:04.835+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:05.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:04.836+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:43:05.496 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:05.452+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:05.496 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:05.452+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:05.496 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:05.452+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:05.496 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:05.453+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:43:06.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:05.751+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:06.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:05.751+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:06.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:05.751+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:06.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:05.752+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:43:06.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:05.837+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:06.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:05.837+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:06.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:05.837+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:06.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:05.838+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:43:06.886 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:43:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:43:06.491+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 24458 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:43:06.886 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:06.717+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:06.887 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:06.717+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:06.887 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:06.717+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:06.887 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:06.717+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:43:06.887 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:06.488+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:06.887 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:06.488+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:06.887 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:06.488+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:06.888 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:06.489+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:43:07.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:06.884+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:07.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:06.884+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:07.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:06.884+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:07.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:06.885+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:43:07.904 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:07.686+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:07.905 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:07.686+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:07.905 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:07.686+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:07.905 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:07.687+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:43:07.905 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:07.453+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:07.906 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:07.453+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:07.906 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:07.453+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:07.906 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:07.454+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:43:08.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:07.903+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:08.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:07.903+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:08.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:07.903+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:08.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:07.904+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:43:08.222 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.146 2024-05-04T12:43:08.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:08.439+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:08.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:08.439+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:08.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:08.439+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:08.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:08.439+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:43:08.699 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:08.642+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:08.699 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:08.642+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:08.700 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:08.642+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:08.700 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:08.643+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:43:09.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:08.922+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:09.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:08.922+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:09.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:08.922+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:09.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:08.923+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:43:09.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:09.487+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:09.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:09.487+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:09.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:09.487+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:09.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:09.488+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:43:09.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:09.605+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:09.699 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:09.605+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:09.699 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:09.605+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:09.699 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:09.606+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:43:10.187 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:09.970+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:10.187 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:09.970+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:10.188 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:09.970+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:10.188 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:09.972+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:43:10.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:10.590+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:10.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:10.590+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:10.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:10.590+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:10.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:10.591+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:43:10.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:10.500+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:10.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:10.500+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:10.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:10.500+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:10.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:10.500+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:43:11.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:11.011+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:11.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:11.011+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:11.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:11.011+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:11.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:11.012+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:43:11.947 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:43:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:43:11.494+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 24547 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:43:11.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:11.511+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:11.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:11.511+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:11.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:11.511+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:11.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:11.512+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:43:11.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:11.594+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:11.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:11.594+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:11.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:11.594+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:11.949 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:11.595+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:43:12.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:11.965+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:12.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:11.965+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:12.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:11.965+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:12.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:11.965+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:43:12.923 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:12.539+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:12.923 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:12.539+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:12.924 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:12.539+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:12.924 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:12.540+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:43:12.924 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:12.627+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:12.924 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:12.627+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:12.924 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:12.627+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:12.924 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:12.628+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:43:13.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:12.921+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:13.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:12.921+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:13.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:12.921+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:13.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:12.923+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:43:13.903 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:13.637+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:13.903 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:13.637+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:13.903 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:13.637+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:13.903 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:13.638+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:43:13.904 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:13.506+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:13.904 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:13.506+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:13.904 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:13.506+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:13.904 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:13.507+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:43:14.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:13.901+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:14.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:13.901+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:14.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:13.901+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:14.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:13.902+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:43:14.669 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:14.510+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:14.669 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:14.510+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:14.669 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:14.510+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:14.669 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:14.511+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:43:14.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:14.887+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:14.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:14.887+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:14.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:14.887+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:14.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:14.888+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:43:14.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:14.667+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:14.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:14.667+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:14.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:14.667+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:14.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:14.668+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:43:15.914 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:15.709+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:15.914 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:15.709+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:15.914 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:15.709+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:15.914 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:15.710+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:43:15.915 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:15.503+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:15.915 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:15.503+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:15.915 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:15.503+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:15.915 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:15.504+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:43:16.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:15.912+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:16.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:15.912+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:16.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:15.912+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:16.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:15.913+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:43:16.941 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:16.553+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:16.941 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:16.553+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:16.941 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:16.553+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:16.941 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:16.554+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:43:16.941 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:16.696+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:16.942 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:16.696+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:16.942 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:16.696+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:16.942 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:16.697+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:43:16.942 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:43:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:43:16.497+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 24640 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:43:17.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:16.939+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:17.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:16.939+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:17.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:16.939+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:17.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:16.940+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:43:17.917 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:17.542+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:17.917 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:17.542+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:17.917 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:17.542+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:17.917 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:17.543+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:43:17.918 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:17.657+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:17.918 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:17.657+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:17.918 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:17.657+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:17.918 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:17.658+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:43:18.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:17.915+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:18.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:17.915+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:18.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:17.915+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:18.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:17.916+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:43:18.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:18.548+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:18.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:18.548+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:18.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:18.548+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:18.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:18.549+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:43:18.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:18.643+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:18.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:18.644+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:18.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:18.644+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:18.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:18.644+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:43:19.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:18.959+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:19.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:18.959+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:19.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:18.959+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:19.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:18.960+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:43:19.832 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:19.562+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:19.833 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:19.562+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:19.833 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:19.562+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:19.833 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:19.563+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:43:19.833 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:19.638+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:19.833 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:19.638+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:19.834 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:19.638+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:19.834 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:19.639+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:43:20.188 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:19.945+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:20.188 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:19.945+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:20.189 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:19.945+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:20.189 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:19.946+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:43:20.223 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-05-04T12:43:20.224 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-04T12:43:20.827 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:20.574+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:20.828 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:20.574+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:20.828 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:20.574+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:20.828 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:20.575+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:43:20.828 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:20.597+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:20.828 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:20.597+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:20.829 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:20.597+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:20.829 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:20.598+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:43:21.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:20.949+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:21.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:20.949+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:21.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:20.949+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:21.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:20.951+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:43:21.947 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:43:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:43:21.499+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 24729 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:43:21.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:21.624+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:21.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:21.624+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:21.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:21.624+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:21.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:21.625+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:43:21.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:21.644+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:21.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:21.644+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:21.949 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:21.644+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:21.949 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:21.645+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:43:22.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:21.970+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:22.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:21.970+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:22.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:21.970+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:22.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:21.971+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:43:22.946 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:22.649+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:22.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:22.649+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:22.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:22.649+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:22.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:22.649+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:43:22.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:22.586+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:22.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:22.586+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:22.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:22.586+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:22.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:22.587+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:43:23.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:22.945+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:23.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:22.945+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:23.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:22.945+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:23.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:22.946+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:43:23.294 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.146 2024-05-04T12:43:23.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:23.628+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:23.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:23.628+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:23.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:23.628+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:23.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:23.629+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:43:23.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:23.674+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:23.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:23.674+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:23.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:23.674+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:23.949 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:23.675+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:43:24.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:23.962+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:24.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:23.962+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:24.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:23.962+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:24.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:23.962+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:43:24.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:24.676+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:24.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:24.676+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:24.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:24.676+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:24.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:24.677+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:43:24.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:24.708+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:24.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:24.708+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:24.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:24.708+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:24.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:24.708+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:43:25.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:24.946+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:25.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:24.946+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:25.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:24.946+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:25.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:24.947+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:43:25.933 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:25.658+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:25.934 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:25.658+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:25.934 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:25.658+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:25.934 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:25.659+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:43:25.934 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:25.682+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:25.934 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:25.682+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:25.935 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:25.682+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:25.935 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:25.683+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:43:26.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:25.931+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:26.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:25.932+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:26.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:25.932+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:26.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:25.932+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:43:26.295 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-05-04T12:43:26.296 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-04T12:43:26.366 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.146 2024-05-04T12:43:26.367 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-05-04T12:43:26.367 INFO:tasks.cephadm:Checking cluster log for badness... 2024-05-04T12:43:26.367 DEBUG:teuthology.orchestra.run.smithi079:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/d9b985c4-0a0e-11ef-bc97-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v OSD_DOWN | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | head -n 1 2024-05-04T12:43:26.405 INFO:tasks.cephadm:Compressing logs... 2024-05-04T12:43:26.405 DEBUG:teuthology.orchestra.run.smithi079:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-05-04T12:43:26.447 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-05-04T12:43:26.448 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-04T12:43:26.468 INFO:teuthology.orchestra.run.smithi079.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/d9b985c4-0a0e-11ef-bc97-c7b262605968/ceph-mds.cephfs.smithi079.pspich.log 2024-05-04T12:43:26.468 INFO:teuthology.orchestra.run.smithi079.stderr:: No such file or directory 2024-05-04T12:43:26.469 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/d9b985c4-0a0e-11ef-bc97-c7b262605968/ceph.log 2024-05-04T12:43:26.469 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/d9b985c4-0a0e-11ef-bc97-c7b262605968/ceph-osd.2.log 2024-05-04T12:43:26.469 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/d9b985c4-0a0e-11ef-bc97-c7b262605968/ceph-mds.cephfs.smithi079.pspich.log: /var/log/ceph/d9b985c4-0a0e-11ef-bc97-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/d9b985c4-0a0e-11ef-bc97-c7b262605968/ceph.audit.log 2024-05-04T12:43:26.469 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/d9b985c4-0a0e-11ef-bc97-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/d9b985c4-0a0e-11ef-bc97-c7b262605968/ceph-osd.1.log 2024-05-04T12:43:26.471 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/d9b985c4-0a0e-11ef-bc97-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/d9b985c4-0a0e-11ef-bc97-c7b262605968/ceph-osd.0.log 2024-05-04T12:43:26.471 INFO:teuthology.orchestra.run.smithi079.stderr: 86.7% -- replaced with /var/log/ceph/d9b985c4-0a0e-11ef-bc97-c7b262605968/ceph.log.gz 2024-05-04T12:43:26.471 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/d9b985c4-0a0e-11ef-bc97-c7b262605968/ceph-mds.cephfs.smithi079.hzcmip.log 2024-05-04T12:43:26.471 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/d9b985c4-0a0e-11ef-bc97-c7b262605968/ceph-osd.1.log: /var/log/ceph/d9b985c4-0a0e-11ef-bc97-c7b262605968/ceph-osd.0.log: /var/log/ceph/d9b985c4-0a0e-11ef-bc97-c7b262605968/ceph-mds.cephfs.smithi079.hzcmip.log: gzip -5 --verbose -- /var/log/ceph/d9b985c4-0a0e-11ef-bc97-c7b262605968/ceph-mon.smithi079.log 2024-05-04T12:43:26.472 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/d9b985c4-0a0e-11ef-bc97-c7b262605968/ceph.cephadm.log 2024-05-04T12:43:26.472 INFO:teuthology.orchestra.run.smithi079.stderr: 90.6% -- replaced with /var/log/ceph/d9b985c4-0a0e-11ef-bc97-c7b262605968/ceph.audit.log.gz 2024-05-04T12:43:26.476 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/d9b985c4-0a0e-11ef-bc97-c7b262605968/ceph-mon.smithi079.log: /var/log/ceph/d9b985c4-0a0e-11ef-bc97-c7b262605968/ceph.cephadm.log: 83.6% -- replaced with /var/log/ceph/d9b985c4-0a0e-11ef-bc97-c7b262605968/ceph.cephadm.log.gz 2024-05-04T12:43:26.477 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/d9b985c4-0a0e-11ef-bc97-c7b262605968/ceph-mgr.smithi079.dgcpwl.log 2024-05-04T12:43:26.477 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/d9b985c4-0a0e-11ef-bc97-c7b262605968/ceph-client.ceph-exporter.smithi079.log 2024-05-04T12:43:26.479 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/d9b985c4-0a0e-11ef-bc97-c7b262605968/ceph-mgr.smithi079.dgcpwl.log: gzip -5 --verbose -- /var/log/ceph/d9b985c4-0a0e-11ef-bc97-c7b262605968/ceph-volume.log 2024-05-04T12:43:26.481 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/d9b985c4-0a0e-11ef-bc97-c7b262605968/ceph-client.ceph-exporter.smithi079.log: 94.5% -- replaced with /var/log/ceph/d9b985c4-0a0e-11ef-bc97-c7b262605968/ceph-client.ceph-exporter.smithi079.log.gz 2024-05-04T12:43:26.491 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-05-04T12:43:26.494 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/d9b985c4-0a0e-11ef-bc97-c7b262605968/ceph-volume.log: 94.8% -- replaced with /var/log/ceph/d9b985c4-0a0e-11ef-bc97-c7b262605968/ceph-mds.cephfs.smithi079.pspich.log.gz 2024-05-04T12:43:26.499 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/cephadm.log: 90.7% -- replaced with /var/log/ceph/d9b985c4-0a0e-11ef-bc97-c7b262605968/ceph-volume.log.gz 2024-05-04T12:43:26.513 INFO:teuthology.orchestra.run.smithi079.stderr: 90.8% -- replaced with /var/log/ceph/cephadm.log.gz 2024-05-04T12:43:26.654 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:43:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:43:26.505+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 24821 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:43:26.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:26.674+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:26.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:26.674+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:26.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:26.674+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:26.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:26.679+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:43:26.949 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:26.652+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:26.949 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:26.652+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:26.949 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:26.654+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:26.950 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:26.654+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:43:27.176 INFO:teuthology.orchestra.run.smithi079.stderr:gzip: /var/log/ceph/d9b985c4-0a0e-11ef-bc97-c7b262605968/ceph-mgr.smithi079.dgcpwl.log: file size changed while zipping 2024-05-04T12:43:27.177 INFO:teuthology.orchestra.run.smithi079.stderr: 89.7% -- replaced with /var/log/ceph/d9b985c4-0a0e-11ef-bc97-c7b262605968/ceph-mgr.smithi079.dgcpwl.log.gz 2024-05-04T12:43:27.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:26.945+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:27.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:26.945+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:27.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:26.945+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:27.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:26.947+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:43:27.479 INFO:teuthology.orchestra.run.smithi079.stderr:gzip: /var/log/ceph/d9b985c4-0a0e-11ef-bc97-c7b262605968/ceph-mon.smithi079.log: file size changed while zipping 2024-05-04T12:43:27.480 INFO:teuthology.orchestra.run.smithi079.stderr: 92.5% -- replaced with /var/log/ceph/d9b985c4-0a0e-11ef-bc97-c7b262605968/ceph-mon.smithi079.log.gz 2024-05-04T12:43:27.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:27.658+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:27.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:27.658+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:27.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:27.658+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:27.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:27.659+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:43:27.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:27.674+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:27.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:27.674+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:27.949 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:27.674+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:27.949 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:27.675+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:43:28.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:27.946+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:28.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:27.946+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:28.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:27.946+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:28.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:27.947+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:43:28.945 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:28.662+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:28.946 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:28.662+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:28.946 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:28.662+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:28.946 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:28.663+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:43:28.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:28.673+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:28.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:28.673+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:28.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:28.673+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:28.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:28.674+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:43:28.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:28.943+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:29.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:28.943+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:29.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:28.943+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:29.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:28.944+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:43:29.506 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.146 2024-05-04T12:43:29.833 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:29.639+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:29.833 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:29.639+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:29.834 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:29.639+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:29.834 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:29.641+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:43:29.834 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:29.665+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:29.834 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:29.665+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:29.834 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:29.665+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:29.835 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:29.666+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:43:30.190 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:29.894+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:30.190 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:29.894+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:30.190 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:29.894+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:30.191 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:29.895+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:43:30.441 INFO:teuthology.orchestra.run.smithi079.stderr:gzip: /var/log/ceph/d9b985c4-0a0e-11ef-bc97-c7b262605968/ceph-osd.2.log: file size changed while zipping 2024-05-04T12:43:30.441 INFO:teuthology.orchestra.run.smithi079.stderr: 95.0% -- replaced with /var/log/ceph/d9b985c4-0a0e-11ef-bc97-c7b262605968/ceph-osd.2.log.gz 2024-05-04T12:43:30.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:30.664+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:30.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:30.664+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:30.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:30.664+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:30.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:30.665+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:43:30.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:30.680+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:30.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:30.680+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:30.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:30.680+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:30.949 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:30.681+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:43:30.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:30.898+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:30.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:30.898+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:30.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:30.898+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:30.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:30.900+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:43:31.012 INFO:teuthology.orchestra.run.smithi079.stderr:gzip: /var/log/ceph/d9b985c4-0a0e-11ef-bc97-c7b262605968/ceph-osd.0.log: file size changed while zipping 2024-05-04T12:43:31.013 INFO:teuthology.orchestra.run.smithi079.stderr: 95.0% -- replaced with /var/log/ceph/d9b985c4-0a0e-11ef-bc97-c7b262605968/ceph-osd.0.log.gz 2024-05-04T12:43:31.871 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:31.626+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:31.871 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:31.626+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:31.872 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:31.626+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:31.872 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:31.627+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:43:31.873 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:43:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:43:31.509+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 24910 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:43:31.874 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:31.638+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:31.874 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:31.638+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:31.874 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:31.638+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:31.875 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:31.639+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:43:32.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:31.870+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:32.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:31.870+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:32.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:31.870+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:32.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:31.871+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:43:32.700 INFO:teuthology.orchestra.run.smithi079.stderr:gzip: /var/log/ceph/d9b985c4-0a0e-11ef-bc97-c7b262605968/ceph-osd.1.log: file size changed while zipping 2024-05-04T12:43:32.701 INFO:teuthology.orchestra.run.smithi079.stderr: 96.3% -- replaced with /var/log/ceph/d9b985c4-0a0e-11ef-bc97-c7b262605968/ceph-osd.1.log.gz 2024-05-04T12:43:32.916 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:32.680+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:32.916 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:32.680+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:32.916 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:32.680+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:32.916 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:32.681+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:43:32.916 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:32.662+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:32.917 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:32.662+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:32.917 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:32.662+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:32.917 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:32.663+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:43:33.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:32.915+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:33.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:32.915+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:33.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:32.915+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:33.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:32.916+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:43:33.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:33.649+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:33.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:33.649+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:33.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:33.649+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:33.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:33.650+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:43:33.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:33.872+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:33.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:33.872+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:33.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:33.872+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:33.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:33.873+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:43:33.949 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:33.715+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:33.949 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:33.715+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:33.949 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:33.715+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:33.949 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:33 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:33.716+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:43:34.850 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:34.602+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:34.850 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:34.602+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:34.850 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:34.602+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:34.850 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:34.603+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:43:34.850 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:34.725+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:34.851 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:34.725+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:34.851 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:34.725+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:34.851 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:34.727+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:43:35.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:34.914+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:35.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:34.914+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:35.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:34.914+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:35.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:34 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:34.915+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:43:35.895 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:35.697+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:35.895 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:35.697+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:35.896 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:35.697+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:35.896 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:35.698+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:43:35.896 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:35.639+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:35.896 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:35.639+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:35.896 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:35.639+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:35.896 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:35.640+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:43:36.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:35.893+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:36.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:35.893+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:36.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:35.893+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:36.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:35 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:35.894+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:43:36.849 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:36.683+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:36.849 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:36.683+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:36.850 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:36.683+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:36.850 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:36.684+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:43:36.850 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:36.742+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:36.850 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:36.742+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:36.850 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:36.742+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:36.851 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:36.743+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:43:36.851 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:43:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:43:36.511+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 25003 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:43:37.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:36.847+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:37.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:36.847+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:37.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:36.847+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:37.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:36 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:36.848+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:43:37.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:37.791+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:37.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:37.791+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:37.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:37.791+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:37.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:37.791+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:43:37.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:37.678+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:37.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:37.678+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:37.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:37.678+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:37.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:37.679+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:43:37.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:37.890+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:37.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:37.890+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:37.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:37.890+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:37.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:37 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:37.891+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:43:38.507 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-05-04T12:43:38.508 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-04T12:43:38.915 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:38.743+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:38.915 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:38.743+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:38.915 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:38.743+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:38.916 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:38.744+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:43:38.916 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:38.639+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:38.916 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:38.639+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:38.916 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:38.639+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:38.916 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:38.639+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:43:39.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:38.913+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:39.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:38.913+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:39.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:38.913+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:39.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:38 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:38.914+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:43:39.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:39.629+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:39.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:39.629+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:39.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:39.629+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:39.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:39.630+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:43:39.949 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:39.696+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:39.949 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:39.696+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:39.949 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:39.696+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:39.950 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:39.698+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:43:40.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:39.948+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:40.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:39.948+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:40.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:39.948+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:40.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:39 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:39.949+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:43:40.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:40.655+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:40.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:40.655+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:40.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:40.655+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:40.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:40.656+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:43:40.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:40.674+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:40.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:40.674+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:40.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:40.674+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:40.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:40.675+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:43:41.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:40.964+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:41.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:40.964+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:41.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:40.964+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:41.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:40 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:40.964+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:43:41.566 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.146 2024-05-04T12:43:41.947 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:43:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:43:41.513+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 25092 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:43:41.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:41.719+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:41.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:41.719+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:41.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:41.719+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:41.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:41.720+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:43:41.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:41.635+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:41.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:41.635+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:41.949 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:41.635+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:41.949 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:41.636+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:43:42.332 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:41.976+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:42.332 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:41.976+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:42.332 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:41.976+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:42.332 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:41 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:41.977+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:43:42.930 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:42.655+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:42.930 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:42.655+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:42.931 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:42.655+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:42.931 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:42.656+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:43:42.931 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:42.696+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:42.931 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:42.696+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:42.931 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:42.696+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:42.931 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:42.697+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:43:43.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:42.928+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:43.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:42.928+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:43.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:42.928+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:43.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:42 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:42.930+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:43:43.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:43.692+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:43.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:43.692+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:43.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:43.692+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:43.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:43.693+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:43:43.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:43.694+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:43.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:43.694+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:43.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:43.694+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:43.949 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:43.695+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:43:43.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:43.898+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:43.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:43.898+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:43.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:43.898+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:43.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:43 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:43.899+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:43:44.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:44.715+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:44.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:44.715+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:44.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:44.715+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:44.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:44.716+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:43:44.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:44.894+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:44.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:44.894+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:44.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:44.894+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:44.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:44.895+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:43:44.949 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:44.675+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:44.949 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:44.675+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:44.949 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:44.675+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:44.949 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:44 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:44.676+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:43:45.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:45.892+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:45.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:45.892+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:45.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:45.892+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:45.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:45.892+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:43:45.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:45.679+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:45.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:45.679+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:45.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:45.679+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:45.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:45.680+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:43:45.949 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:45.698+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:45.949 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:45.698+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:45.949 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:45.698+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:45.949 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:45 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:45.699+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:43:46.854 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:46.681+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:46.854 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:46.681+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:46.854 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:46.681+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:46.854 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:46.683+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:43:46.855 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:46.737+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:46.855 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:46.737+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:46.855 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:46.737+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:46.855 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:46.739+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:43:46.855 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:43:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:43:46.516+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 25183 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:43:46.863 INFO:teuthology.orchestra.run.smithi079.stderr: 91.9% -- replaced with /var/log/ceph/d9b985c4-0a0e-11ef-bc97-c7b262605968/ceph-mds.cephfs.smithi079.hzcmip.log.gz 2024-05-04T12:43:46.865 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-05-04T12:43:46.866 INFO:teuthology.orchestra.run.smithi079.stderr:real 0m20.407s 2024-05-04T12:43:46.866 INFO:teuthology.orchestra.run.smithi079.stderr:user 0m35.838s 2024-05-04T12:43:46.866 INFO:teuthology.orchestra.run.smithi079.stderr:sys 0m0.948s 2024-05-04T12:43:47.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:46.852+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:47.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:46.852+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:47.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:46.852+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:47.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:46 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:46.853+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:43:48.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:47.762+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:48.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:47.762+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:48.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:47.762+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:48.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:47.763+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:43:48.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:47.731+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:48.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:47.731+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:48.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:47.731+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:48.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:47.732+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:43:48.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:47.861+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:48.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:47.861+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:48.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:47.861+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:48.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:47 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:47.862+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:43:49.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:48.745+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:49.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:48.745+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:49.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:48.745+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:49.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:48.746+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:43:49.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:48.783+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:49.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:48.783+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:49.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:48.783+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:49.199 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:48.783+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:43:49.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:48.849+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:49.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:48.849+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:49.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:48.849+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:49.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:48 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:48.850+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:43:50.191 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:49.773+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:50.192 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:49.773+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:50.192 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:49.773+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:50.192 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:49.774+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:43:50.192 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:49.824+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:50.193 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:49.824+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:50.193 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:49.824+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:50.193 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:49.825+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:43:50.193 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:49.749+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:50.193 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:49.749+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:50.193 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:49.749+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:50.193 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:49 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:49.750+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:43:51.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:50.825+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:51.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:50.825+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:51.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:50.825+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:51.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:50.826+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:43:51.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:50.759+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:51.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:50.759+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:51.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:50.759+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:51.199 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:50.760+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:43:51.199 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:50.747+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:51.199 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:50.747+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:51.199 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:50.747+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:51.199 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:50 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:50.748+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:43:51.785 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:43:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:43:51.519+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 25272 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:43:51.785 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:51.759+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:51.786 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:51.759+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:51.786 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:51.759+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:51.786 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:51.760+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:43:52.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:51.783+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:52.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:51.783+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:52.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:51.783+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:52.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:51.785+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:43:52.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:51.806+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:52.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:51.806+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:52.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:51.806+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:52.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:51 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:51.807+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:43:53.197 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:52.830+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:53.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:52.830+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:53.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:52.830+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:53.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:52.831+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:43:53.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:52.748+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:53.199 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:52.748+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:53.199 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:52.748+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:53.199 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:52.749+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:43:53.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:52.824+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:53.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:52.824+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:53.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:52.824+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:53.200 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:52 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:52.825+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:43:53.567 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-05-04T12:43:53.568 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-04T12:43:54.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:53.870+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:54.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:53.870+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:54.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:53.870+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:54.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:53.871+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:43:54.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:53.711+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:54.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:53.712+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:54.199 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:53.712+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:54.199 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:53.713+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:43:54.199 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:53.791+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:54.199 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:53.791+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:54.199 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:53.791+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:54.199 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:53 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:53.792+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:43:55.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:54.718+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:55.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:54.718+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:55.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:54.718+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:55.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:54.719+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:43:55.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:54.776+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:55.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:54.776+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:55.199 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:54.776+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:55.199 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:54.776+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:43:55.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:54.837+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:55.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:54.837+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:55.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:54.837+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:55.199 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:54 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:54.838+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:43:56.197 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:55.750+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:56.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:55.750+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:56.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:55.750+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:56.198 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:55.750+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:43:56.199 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:55.734+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:56.199 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:55.734+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:56.200 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:55.734+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:56.200 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:55.734+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:43:56.200 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:55.866+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:56.200 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:55.866+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:56.200 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:55.866+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:56.200 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:55 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:55.866+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:43:56.638 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.146 2024-05-04T12:43:56.782 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:43:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:43:56.522+0000 7fda54d34700 -1 mon.smithi079@0(probing) e2 get_health_metrics reporting 25365 slow ops, oldest is monmgrreport(gid 14219, 1 checks, 1 progress events) 2024-05-04T12:43:56.783 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:56.705+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:56.783 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:56.705+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:56.783 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:56.705+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:56.783 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:56.706+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:43:57.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:56.891+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:57.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:56.891+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:57.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:56.891+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:57.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:56.891+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:43:57.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:56.780+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:57.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:56.780+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:57.199 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:56.780+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:57.199 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:56 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:56.781+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:43:57.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:57.663+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:57.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:57.663+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:57.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:57.663+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:57.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:57.664+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:43:57.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:57.897+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:57.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:57.897+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:57.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:57.897+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:57.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:57.898+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:43:57.949 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:57.746+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:57.949 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:57.746+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:57.949 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:57.746+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:57.949 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:57 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:57.747+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:43:58.909 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:58.781+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:58.909 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:58.781+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:58.909 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:58.781+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:58.910 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:58.782+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:43:58.910 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:58.621+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:58.910 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:58.621+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:58.910 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:58.621+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:58.910 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:58.621+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:43:59.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:58.907+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:59.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:58.907+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:59.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:58.907+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:59.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:58 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:58.908+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:43:59.639 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-05-04T12:43:59.640 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-04T12:43:59.714 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.146 2024-05-04T12:43:59.714 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-05-04T12:43:59.715 INFO:tasks.cephadm:Removing cluster... 2024-05-04T12:43:59.715 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid d9b985c4-0a0e-11ef-bc97-c7b262605968 --force 2024-05-04T12:43:59.879 INFO:teuthology.orchestra.run.smithi079.stdout:Deleting cluster with fsid: d9b985c4-0a0e-11ef-bc97-c7b262605968 2024-05-04T12:43:59.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:59.863+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:59.947 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:59.863+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:59.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:59.863+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:43:59.948 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:43:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:43:59.864+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:43:59.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:59.640+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:59.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:59.640+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:59.948 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:59.640+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:43:59.949 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:43:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:43:59.641+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:43:59.949 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:59.756+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:59.949 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:59.756+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:59.949 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:59.756+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:43:59.949 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:43:59 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:43:59.757+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:44:00.261 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:43:59 smithi079 systemd[1]: Stopping Ceph mon.smithi079 for d9b985c4-0a0e-11ef-bc97-c7b262605968... 2024-05-04T12:44:00.526 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:44:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:44:00.260+0000 7fda57d3a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi079 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-05-04T12:44:00.526 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:44:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079[27218]: 2024-05-04T12:44:00.260+0000 7fda57d3a700 -1 mon.smithi079@0(probing) e2 *** Got Signal Terminated *** 2024-05-04T12:44:00.831 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:44:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:44:00.742+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:44:00.831 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:44:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:44:00.742+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:44:00.832 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:44:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:44:00.742+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:44:00.832 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:44:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:44:00.742+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:44:00.832 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:44:00 smithi079 podman[82871]: 2024-05-04 12:44:00.525368052 +0000 UTC m=+0.390774908 container died ce54f11f45dfb16f4c6b4666f6ce4e5e271e8dc289d0e9051cc67cd411322a92 (image=quay.io/ceph/ceph:v18.2.1, name=ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079, GIT_CLEAN=True, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, ceph=True, io.buildah.version=1.29.1, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux ) 2024-05-04T12:44:00.832 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:44:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:44:00.607+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:44:00.833 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:44:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:44:00.607+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:44:00.833 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:44:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:44:00.607+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:44:00.833 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:44:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:44:00.608+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:44:01.165 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:44:00.829+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:44:01.165 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:44:00.829+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:44:01.165 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:44:00.829+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:44:01.166 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:00 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:44:00.830+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:44:01.166 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:44:00 smithi079 podman[82871]: 2024-05-04 12:44:00.937821442 +0000 UTC m=+0.803228296 container cleanup ce54f11f45dfb16f4c6b4666f6ce4e5e271e8dc289d0e9051cc67cd411322a92 (image=quay.io/ceph/ceph:v18.2.1, name=ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=HEAD, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.29.1) 2024-05-04T12:44:01.166 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:44:00 smithi079 bash[82871]: ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079 2024-05-04T12:44:01.421 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:44:01 smithi079 podman[82883]: 2024-05-04 12:44:01.164707237 +0000 UTC m=+0.637301466 container remove ce54f11f45dfb16f4c6b4666f6ce4e5e271e8dc289d0e9051cc67cd411322a92 (image=quay.io/ceph/ceph:v18.2.1, name=ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-mon-smithi079, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, RELEASE=HEAD, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, GIT_BRANCH=HEAD) 2024-05-04T12:44:01.697 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:44:01 smithi079 systemd[1]: ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968@mon.smithi079.service: Deactivated successfully. 2024-05-04T12:44:01.698 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:44:01 smithi079 systemd[1]: Stopped Ceph mon.smithi079 for d9b985c4-0a0e-11ef-bc97-c7b262605968. 2024-05-04T12:44:01.698 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:May 04 12:44:01 smithi079 systemd[1]: ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968@mon.smithi079.service: Consumed 20.424s CPU time. 2024-05-04T12:44:01.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:44:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:44:01.562+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:44:01.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:44:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:44:01.562+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:44:01.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:44:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:44:01.562+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:44:01.699 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:44:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:44:01.563+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:44:02.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:44:01.874+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:44:02.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:44:01.874+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:44:02.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:44:01.874+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:44:02.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:44:01.875+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:44:02.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:44:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:44:01.784+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:44:02.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:44:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:44:01.784+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:44:02.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:44:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:44:01.784+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:44:02.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:44:01 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:44:01.786+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:44:02.648 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:44:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:44:02.515+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:44:02.648 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:44:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:44:02.515+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:44:02.649 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:44:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:44:02.515+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:44:02.649 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:44:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:44:02.516+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:44:02.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:44:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:44:02.746+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:44:02.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:44:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:44:02.746+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:44:02.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:44:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:44:02.746+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:44:02.949 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:44:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:44:02.747+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:44:02.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:44:02.853+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:44:02.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:44:02.853+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:44:02.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:44:02.853+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:44:02.949 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:02 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:44:02.853+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:44:03.579 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:44:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:44:03.467+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:44:03.579 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:44:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:44:03.467+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:44:03.579 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:44:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:44:03.467+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:44:03.579 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:44:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:44:03.468+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:44:03.888 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:44:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:44:03.755+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:44:03.889 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:44:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:44:03.755+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:44:03.889 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:44:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:44:03.755+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:44:03.889 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:44:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:44:03.756+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:44:04.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:44:03.887+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:44:04.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:44:03.887+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:44:04.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:44:03.887+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:44:04.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:03 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:44:03.888+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:44:04.760 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:44:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:44:04.461+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:44:04.760 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:44:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:44:04.461+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:44:04.760 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:44:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:44:04.461+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:44:04.760 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:44:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:44:04.462+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:44:05.077 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:44:04.899+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:44:05.077 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:44:04.899+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:44:05.077 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:44:04.899+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:44:05.077 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:44:04.900+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:44:05.077 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:44:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:44:04.758+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:44:05.078 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:44:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:44:04.758+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:44:05.078 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:44:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:44:04.758+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:44:05.078 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:44:04 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:44:04.759+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:44:05.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:44:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:44:05.452+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:44:05.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:44:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:44:05.452+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:44:05.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:44:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:44:05.452+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:44:05.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:44:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:44:05.453+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:44:05.973 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:44:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:44:05.713+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:44:05.974 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:44:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:44:05.713+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:44:05.974 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:44:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:44:05.713+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:44:05.974 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:44:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:44:05.714+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:44:05.974 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:44:05.947+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:44:05.974 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:44:05.947+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:44:05.975 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:44:05.947+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:44:05.975 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:05 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:44:05.948+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:44:06.677 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:44:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:44:06.428+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:44:06.678 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:44:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:44:06.428+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:44:06.678 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:44:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:44:06.428+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:44:06.678 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:44:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:44:06.429+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:44:06.936 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:44:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:44:06.751+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:44:06.936 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:44:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:44:06.751+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:44:06.937 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:44:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:44:06.751+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:44:06.937 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:44:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:44:06.752+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:44:07.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:44:06.992+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:44:07.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:44:06.992+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:44:07.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:44:06.992+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:44:07.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:06 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:44:06.993+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:44:07.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:44:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:44:07.402+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:44:07.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:44:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:44:07.402+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:44:07.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:44:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:44:07.402+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:44:07.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:44:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:44:07.402+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:44:08.052 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:44:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:44:07.756+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:44:08.052 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:44:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:44:07.756+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:44:08.053 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:44:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:44:07.756+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:44:08.053 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:44:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:44:07.757+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:44:08.053 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:44:07.983+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:44:08.054 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:44:07.983+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:44:08.054 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:44:07.983+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:44:08.054 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:07 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:44:07.984+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:44:08.697 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:44:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:44:08.427+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:44:08.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:44:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:44:08.427+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:44:08.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:44:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:44:08.427+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:44:08.698 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:44:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:44:08.428+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:44:09.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:44:08.941+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:44:09.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:44:08.941+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:44:09.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:44:08.941+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:44:09.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:44:08.942+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:44:09.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:44:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:44:08.767+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:44:09.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:44:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:44:08.767+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:44:09.198 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:44:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:44:08.767+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:44:09.199 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:44:08 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:44:08.768+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:44:09.640 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:44:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:44:09.392+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:44:09.640 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:44:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:44:09.392+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:44:09.641 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:44:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:44:09.392+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:44:09.641 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:44:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:44:09.393+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:44:09.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:44:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:44:09.719+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:44:09.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:44:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:44:09.719+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:44:09.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:44:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:44:09.719+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:44:09.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:44:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:44:09.720+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:44:10.280 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:44:09.946+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:44:10.280 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:44:09.946+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:44:10.280 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:44:09.946+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:44:10.280 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:09 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:44:09.947+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:44:10.590 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:44:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:44:10.365+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:44:10.590 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:44:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:44:10.365+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:44:10.590 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:44:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:44:10.365+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:44:10.590 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:44:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:44:10.365+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:44:10.933 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:44:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:44:10.707+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:44:10.933 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:44:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:44:10.707+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:44:10.934 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:44:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:44:10.707+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:44:10.934 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:44:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:44:10.708+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:44:11.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:44:10.990+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:44:11.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:44:10.990+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:44:11.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:44:10.990+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:44:11.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:10 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:44:10.990+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:44:11.512 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:44:11 smithi079 systemd[1]: Stopping Ceph osd.0 for d9b985c4-0a0e-11ef-bc97-c7b262605968... 2024-05-04T12:44:11.513 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:44:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:44:11.370+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:44:11.513 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:44:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:44:11.370+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:44:11.513 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:44:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:44:11.370+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:44:11.514 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:44:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:44:11.370+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:44:11.514 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:44:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:44:11.511+0000 7f234e985700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-04T12:44:11.514 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:44:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:44:11.511+0000 7f234e985700 -1 osd.0 46 *** Got signal Terminated *** 2024-05-04T12:44:11.514 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:44:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:44:11.511+0000 7f234e985700 -1 osd.0 46 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-04T12:44:11.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:44:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:44:11.735+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:44:11.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:44:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:44:11.735+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:44:11.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:44:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:44:11.735+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:44:11.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:44:11 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:44:11.736+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:44:12.415 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:44:12.014+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:44:12.416 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:44:12.014+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:44:12.416 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:44:12.014+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:44:12.416 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:44:12.015+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:44:12.692 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:44:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:44:12.414+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:44:12.692 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:44:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:44:12.414+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:44:12.692 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:44:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:44:12.414+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:44:12.693 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:44:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:44:12.415+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:44:12.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:44:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:44:12.690+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:44:12.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:44:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:44:12.690+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:44:12.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:44:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:44:12.690+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:44:12.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:44:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:44:12.691+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:44:13.386 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:44:12.994+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:44:13.387 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:44:12.994+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:44:13.387 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:44:12.994+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:44:13.387 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:12 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:44:12.994+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:44:13.660 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:44:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:44:13.385+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:44:13.660 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:44:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:44:13.385+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:44:13.661 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:44:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:44:13.385+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:44:13.661 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:44:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:44:13.386+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:44:13.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:44:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:44:13.659+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:44:13.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:44:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:44:13.659+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:44:13.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:44:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:44:13.659+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:44:13.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:44:13 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:44:13.659+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:44:14.369 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:44:14.014+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:44:14.369 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:44:14.014+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:44:14.370 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:44:14.014+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:44:14.370 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:44:14.015+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:44:14.654 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:44:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:44:14.368+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:44:14.654 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:44:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:44:14.368+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:44:14.654 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:44:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:44:14.368+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:44:14.654 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:44:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:44:14.369+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:44:14.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:44:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:44:14.652+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:44:14.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:44:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:44:14.652+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:44:14.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:44:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:44:14.652+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:44:14.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:44:14 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:44:14.653+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:44:15.390 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:44:15.035+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:44:15.390 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:44:15.035+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:44:15.391 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:44:15.035+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:44:15.391 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:44:15.035+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:44:15.678 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:44:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:44:15.389+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:44:15.678 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:44:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:44:15.389+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:44:15.678 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:44:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:44:15.389+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:44:15.679 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:44:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:44:15.389+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:44:15.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:44:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:44:15.676+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:44:15.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:44:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:44:15.677+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:44:15.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:44:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:44:15.677+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:44:15.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:44:15 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:44:15.677+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:44:16.414 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:44:16.015+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:44:16.414 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:44:16.015+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:44:16.414 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:44:16.015+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:44:16.414 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:44:16.017+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:44:16.682 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:44:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:44:16.413+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:44:16.682 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:44:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:44:16.413+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:44:16.682 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:44:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:44:16.413+0000 7f2347774700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:11.443748+0000 (oldest deadline 2024-05-04T12:38:31.443748+0000) 2024-05-04T12:44:16.683 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:44:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0[46050]: 2024-05-04T12:44:16.413+0000 7f2347774700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1514 3.1 3:83d37751:::100000025d3.00000000:head [write 0~2460 in=2460b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:44:16.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:44:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:44:16.680+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:44:16.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:44:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:44:16.680+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:44:16.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:44:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:44:16.680+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:44:16.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:44:16 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:44:16.681+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:44:17.389 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:44:17.033+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:44:17.389 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:44:17.033+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:44:17.389 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:44:17.033+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:44:17.389 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:44:17.034+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:44:17.390 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:44:17 smithi079 podman[83832]: 2024-05-04 12:44:17.093522703 +0000 UTC m=+5.702273604 container died ee518ff3d7659d8ae331116abf8e296ba13b2caba61294084e07609803f71a96 (image=quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3, name=ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=-18.2.1, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1) 2024-05-04T12:44:17.651 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:44:17 smithi079 podman[83832]: 2024-05-04 12:44:17.495963188 +0000 UTC m=+6.104714097 container cleanup ee518ff3d7659d8ae331116abf8e296ba13b2caba61294084e07609803f71a96 (image=quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3, name=ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.29.1, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, RELEASE=HEAD, CEPH_POINT_RELEASE=-18.2.1) 2024-05-04T12:44:17.651 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:44:17 smithi079 bash[83832]: ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0 2024-05-04T12:44:17.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:44:17 smithi079 podman[83851]: 2024-05-04 12:44:17.739882235 +0000 UTC m=+0.649149248 container remove ee518ff3d7659d8ae331116abf8e296ba13b2caba61294084e07609803f71a96 (image=quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3, name=ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0, maintainer=Guillaume Abrioux , GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20240222, RELEASE=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.29.1) 2024-05-04T12:44:17.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:44:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:44:17.650+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:44:17.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:44:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:44:17.650+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:44:17.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:44:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:44:17.650+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:44:17.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:44:17 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:44:17.651+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:44:18.353 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:44:18.047+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:44:18.353 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:44:18.047+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:44:18.353 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:44:18.047+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:44:18.353 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:44:18.048+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:44:18.353 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:44:18 smithi079 podman[83944]: 2024-05-04 12:44:17.976678704 +0000 UTC m=+0.019713746 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3 2024-05-04T12:44:18.354 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:44:18 smithi079 podman[83944]: 2024-05-04 12:44:18.099298719 +0000 UTC m=+0.142333719 container create 1b78fd7bd0c59601c3d858c59f65ca3e822f8b03385eefe5ea27d1970cd00663 (image=quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3, name=ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0-deactivate, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.29.1, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1) 2024-05-04T12:44:18.354 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:44:18 smithi079 podman[83944]: 2024-05-04 12:44:18.352567717 +0000 UTC m=+0.395602743 container init 1b78fd7bd0c59601c3d858c59f65ca3e822f8b03385eefe5ea27d1970cd00663 (image=quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3, name=ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0-deactivate, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, CEPH_POINT_RELEASE=-18.2.1, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True) 2024-05-04T12:44:18.616 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:44:18 smithi079 podman[83944]: 2024-05-04 12:44:18.356101231 +0000 UTC m=+0.399136236 container start 1b78fd7bd0c59601c3d858c59f65ca3e822f8b03385eefe5ea27d1970cd00663 (image=quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3, name=ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0-deactivate, org.label-schema.vendor=CentOS, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=-18.2.1) 2024-05-04T12:44:18.616 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:44:18 smithi079 podman[83944]: 2024-05-04 12:44:18.461029003 +0000 UTC m=+0.504064019 container attach 1b78fd7bd0c59601c3d858c59f65ca3e822f8b03385eefe5ea27d1970cd00663 (image=quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3, name=ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=HEAD, GIT_BRANCH=HEAD, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git) 2024-05-04T12:44:18.947 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:44:18 smithi079 podman[83944]: 2024-05-04 12:44:18.692838024 +0000 UTC m=+0.735873025 container died 1b78fd7bd0c59601c3d858c59f65ca3e822f8b03385eefe5ea27d1970cd00663 (image=quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3, name=ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0-deactivate, RELEASE=HEAD, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-05-04T12:44:18.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:44:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:44:18.692+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:44:18.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:44:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:44:18.692+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:44:18.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:44:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:44:18.692+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:44:18.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:44:18 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:44:18.693+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:44:19.308 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:44:19.059+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:44:19.309 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:44:19.059+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:44:19.309 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:44:19.059+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:44:19.309 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:44:19.060+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:44:19.677 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:44:19 smithi079 podman[83972]: 2024-05-04 12:44:19.308157914 +0000 UTC m=+0.609583699 container remove 1b78fd7bd0c59601c3d858c59f65ca3e822f8b03385eefe5ea27d1970cd00663 (image=quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3, name=ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-0-deactivate, RELEASE=HEAD, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-18.2.1) 2024-05-04T12:44:19.677 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:44:19 smithi079 systemd[1]: ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968@osd.0.service: Deactivated successfully. 2024-05-04T12:44:19.678 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:44:19 smithi079 systemd[1]: Stopped Ceph osd.0 for d9b985c4-0a0e-11ef-bc97-c7b262605968. 2024-05-04T12:44:19.678 INFO:journalctl@ceph.osd.0.smithi079.stdout:May 04 12:44:19 smithi079 systemd[1]: ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968@osd.0.service: Consumed 13min 23.917s CPU time. 2024-05-04T12:44:19.678 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:44:19 smithi079 systemd[1]: Stopping Ceph osd.1 for d9b985c4-0a0e-11ef-bc97-c7b262605968... 2024-05-04T12:44:19.678 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:44:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:44:19.676+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:44:19.678 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:44:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:44:19.676+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:44:19.679 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:44:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:44:19.676+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:44:19.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:44:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:44:19.677+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:44:19.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:44:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:44:19.828+0000 7f4daddea700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-04T12:44:19.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:44:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:44:19.828+0000 7f4daddea700 -1 osd.1 46 *** Got signal Terminated *** 2024-05-04T12:44:19.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:44:19 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:44:19.828+0000 7f4daddea700 -1 osd.1 46 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-04T12:44:20.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:44:20.055+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:44:20.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:44:20.055+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:44:20.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:44:20.056+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:44:20.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:44:20.056+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:44:21.089 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:44:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:44:20.707+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:44:21.089 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:44:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:44:20.707+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:44:21.090 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:44:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:44:20.707+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:44:21.090 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:44:20 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:44:20.708+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:44:21.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:44:21.088+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:44:21.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:44:21.088+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:44:21.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:44:21.088+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:44:21.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:44:21.088+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:44:22.125 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:44:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:44:21.705+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:44:22.126 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:44:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:44:21.705+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:44:22.126 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:44:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:44:21.705+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:44:22.126 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:44:21 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:44:21.706+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:44:22.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:44:22.124+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:44:22.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:44:22.124+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:44:22.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:44:22.124+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:44:22.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:44:22.125+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:44:23.092 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:44:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:44:22.702+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:44:23.092 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:44:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:44:22.702+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:44:23.093 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:44:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:44:22.702+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:44:23.093 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:44:22 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:44:22.703+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:44:23.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:44:23.090+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:44:23.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:44:23.090+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:44:23.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:44:23.090+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:44:23.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:44:23.092+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:44:23.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:44:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:44:23.664+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:44:23.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:44:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:44:23.664+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:44:23.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:44:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:44:23.664+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:44:23.948 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:44:23 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:44:23.664+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:44:24.449 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:44:24.056+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:44:24.450 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:44:24.056+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:44:24.450 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:44:24.056+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:44:24.450 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:44:24.057+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:44:25.081 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:44:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:44:24.701+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:44:25.081 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:44:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:44:24.701+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:44:25.082 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:44:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:44:24.701+0000 7f4daabe1700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:37:56.039547+0000 (oldest deadline 2024-05-04T12:38:16.039547+0000) 2024-05-04T12:44:25.082 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:44:24 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1[53710]: 2024-05-04T12:44:24.702+0000 7f4daabe1700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.11:561 2.a 2:545c15c7:::200.00000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-05-04T12:44:25.447 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:44:25 smithi079 podman[84059]: 2024-05-04 12:44:25.301915117 +0000 UTC m=+5.603834839 container died 57cfd8dc67ddcf247143faaa203f2a38c84742f3f41a8e2558d11d88d721599b (image=quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3, name=ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , RELEASE=HEAD, io.buildah.version=1.29.1, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.build-date=20240222) 2024-05-04T12:44:25.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:44:25.079+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:44:25.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:44:25.079+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:44:25.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:44:25.079+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:44:25.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:25 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:44:25.080+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:44:25.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:44:25 smithi079 podman[84059]: 2024-05-04 12:44:25.703610607 +0000 UTC m=+6.005530329 container cleanup 57cfd8dc67ddcf247143faaa203f2a38c84742f3f41a8e2558d11d88d721599b (image=quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3, name=ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.29.1, CEPH_POINT_RELEASE=-18.2.1, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20240222, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=HEAD) 2024-05-04T12:44:25.947 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:44:25 smithi079 bash[84059]: ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1 2024-05-04T12:44:26.249 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:44:25 smithi079 podman[84080]: 2024-05-04 12:44:25.94719578 +0000 UTC m=+0.644180589 container remove 57cfd8dc67ddcf247143faaa203f2a38c84742f3f41a8e2558d11d88d721599b (image=quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3, name=ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, RELEASE=HEAD, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git) 2024-05-04T12:44:26.250 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:44:26.118+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:44:26.250 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:44:26.118+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:44:26.250 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:44:26.118+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:44:26.250 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:26 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:44:26.119+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:44:26.527 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:44:26 smithi079 podman[84168]: 2024-05-04 12:44:26.152674824 +0000 UTC m=+0.018063400 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3 2024-05-04T12:44:26.527 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:44:26 smithi079 podman[84168]: 2024-05-04 12:44:26.293055714 +0000 UTC m=+0.158444285 container create b8bf4bd92e4b7a3a98244d58da68d28cb741bc77a5ac009503a9f9b1d48d08d5 (image=quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3, name=ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1-deactivate, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, RELEASE=HEAD, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS) 2024-05-04T12:44:26.817 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:44:26 smithi079 podman[84168]: 2024-05-04 12:44:26.526845957 +0000 UTC m=+0.392234534 container init b8bf4bd92e4b7a3a98244d58da68d28cb741bc77a5ac009503a9f9b1d48d08d5 (image=quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3, name=ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1-deactivate, ceph=True, RELEASE=HEAD, CEPH_POINT_RELEASE=-18.2.1, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1) 2024-05-04T12:44:26.818 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:44:26 smithi079 podman[84168]: 2024-05-04 12:44:26.530415863 +0000 UTC m=+0.395804435 container start b8bf4bd92e4b7a3a98244d58da68d28cb741bc77a5ac009503a9f9b1d48d08d5 (image=quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3, name=ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1-deactivate, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0) 2024-05-04T12:44:26.818 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:44:26 smithi079 podman[84168]: 2024-05-04 12:44:26.634702224 +0000 UTC m=+0.500090787 container attach b8bf4bd92e4b7a3a98244d58da68d28cb741bc77a5ac009503a9f9b1d48d08d5 (image=quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3, name=ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1-deactivate, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=HEAD, io.buildah.version=1.29.1, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) 2024-05-04T12:44:27.130 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:44:26 smithi079 podman[84168]: 2024-05-04 12:44:26.817397925 +0000 UTC m=+0.682786482 container died b8bf4bd92e4b7a3a98244d58da68d28cb741bc77a5ac009503a9f9b1d48d08d5 (image=quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3, name=ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1-deactivate, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240222, RELEASE=HEAD, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-04T12:44:27.423 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:44:27.163+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:44:27.423 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:44:27.163+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:44:27.423 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:44:27.163+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:44:27.423 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:44:27.163+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:44:27.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:27 smithi079 systemd[1]: Stopping Ceph osd.2 for d9b985c4-0a0e-11ef-bc97-c7b262605968... 2024-05-04T12:44:27.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:44:27 smithi079 podman[84195]: 2024-05-04 12:44:27.422654046 +0000 UTC m=+0.596248806 container remove b8bf4bd92e4b7a3a98244d58da68d28cb741bc77a5ac009503a9f9b1d48d08d5 (image=quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3, name=ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-1-deactivate, GIT_BRANCH=HEAD, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240222) 2024-05-04T12:44:27.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:44:27 smithi079 systemd[1]: ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968@osd.1.service: Deactivated successfully. 2024-05-04T12:44:27.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:44:27 smithi079 systemd[1]: Stopped Ceph osd.1 for d9b985c4-0a0e-11ef-bc97-c7b262605968. 2024-05-04T12:44:27.698 INFO:journalctl@ceph.osd.1.smithi079.stdout:May 04 12:44:27 smithi079 systemd[1]: ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968@osd.1.service: Consumed 26min 24.692s CPU time. 2024-05-04T12:44:28.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:44:27.924+0000 7fe0d0871700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-04T12:44:28.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:44:27.924+0000 7fe0d0871700 -1 osd.2 46 *** Got signal Terminated *** 2024-05-04T12:44:28.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:27 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:44:27.925+0000 7fe0d0871700 -1 osd.2 46 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-04T12:44:28.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:44:28.145+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:44:28.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:44:28.145+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:44:28.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:44:28.145+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:44:28.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:28 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:44:28.146+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:44:29.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:44:29.145+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:44:29.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:44:29.145+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:44:29.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:44:29.145+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:44:29.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:29 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:44:29.146+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:44:30.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:44:30.170+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:44:30.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:44:30.170+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:44:30.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:44:30.170+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:44:30.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:30 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:44:30.171+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:44:31.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:44:31.170+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:44:31.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:44:31.170+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:44:31.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:44:31.170+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:44:31.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:31 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:44:31.171+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:44:32.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:44:32.127+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6804 osd.3 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:44:32.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:44:32.127+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6812 osd.4 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:44:32.447 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:44:32.127+0000 7fe0cd668700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.146:6820 osd.5 ever on either front or back, first ping sent 2024-05-04T12:38:01.631037+0000 (oldest deadline 2024-05-04T12:38:21.631037+0000) 2024-05-04T12:44:32.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:32 smithi079 ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2[61382]: 2024-05-04T12:44:32.128+0000 7fe0cd668700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:1558 3.8 3:135f234d:::1000000243e.00000000:head [write 0~2557 in=2557b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-05-04T12:44:33.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:33 smithi079 podman[84273]: 2024-05-04 12:44:33.37735225 +0000 UTC m=+5.574025901 container died 4778a780462b8ef8d2b830bcd8995f2362076668b1668bd6024c1573274e7ba4 (image=quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3, name=ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2, io.buildah.version=1.29.1, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True, CEPH_POINT_RELEASE=-18.2.1, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD) 2024-05-04T12:44:34.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:33 smithi079 podman[84273]: 2024-05-04 12:44:33.700476239 +0000 UTC m=+5.897149931 container cleanup 4778a780462b8ef8d2b830bcd8995f2362076668b1668bd6024c1573274e7ba4 (image=quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3, name=ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux , GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-04T12:44:34.197 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:33 smithi079 bash[84273]: ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2 2024-05-04T12:44:34.198 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:33 smithi079 podman[84294]: 2024-05-04 12:44:33.910152594 +0000 UTC m=+0.530803781 container remove 4778a780462b8ef8d2b830bcd8995f2362076668b1668bd6024c1573274e7ba4 (image=quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3, name=ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2, org.label-schema.schema-version=1.0, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS, io.buildah.version=1.29.1, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=-18.2.1, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True) 2024-05-04T12:44:34.448 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:34 smithi079 podman[84384]: 2024-05-04 12:44:34.215189997 +0000 UTC m=+0.018796228 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3 2024-05-04T12:44:34.449 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:34 smithi079 podman[84384]: 2024-05-04 12:44:34.319408504 +0000 UTC m=+0.123014717 container create 4bec711ec9f63dcdb2079dd466507326eb8377c0e3078f6dcd3772d7aee59fb0 (image=quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3, name=ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2-deactivate, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.29.1) 2024-05-04T12:44:34.869 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:34 smithi079 podman[84384]: 2024-05-04 12:44:34.573238007 +0000 UTC m=+0.376844223 container init 4bec711ec9f63dcdb2079dd466507326eb8377c0e3078f6dcd3772d7aee59fb0 (image=quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3, name=ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2-deactivate, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, GIT_CLEAN=True, RELEASE=HEAD) 2024-05-04T12:44:34.869 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:34 smithi079 podman[84384]: 2024-05-04 12:44:34.576540293 +0000 UTC m=+0.380146508 container start 4bec711ec9f63dcdb2079dd466507326eb8377c0e3078f6dcd3772d7aee59fb0 (image=quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3, name=ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2-deactivate, org.label-schema.schema-version=1.0, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.29.1, RELEASE=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True) 2024-05-04T12:44:34.869 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:34 smithi079 podman[84384]: 2024-05-04 12:44:34.656536933 +0000 UTC m=+0.460143175 container attach 4bec711ec9f63dcdb2079dd466507326eb8377c0e3078f6dcd3772d7aee59fb0 (image=quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3, name=ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2-deactivate, RELEASE=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.1, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS, io.buildah.version=1.29.1, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, GIT_REPO=https://github.com/ceph/ceph-container.git) 2024-05-04T12:44:35.176 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:34 smithi079 podman[84384]: 2024-05-04 12:44:34.869154555 +0000 UTC m=+0.672760763 container died 4bec711ec9f63dcdb2079dd466507326eb8377c0e3078f6dcd3772d7aee59fb0 (image=quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3, name=ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2-deactivate, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1) 2024-05-04T12:44:35.697 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:35 smithi079 podman[84421]: 2024-05-04 12:44:35.426725344 +0000 UTC m=+0.552232931 container remove 4bec711ec9f63dcdb2079dd466507326eb8377c0e3078f6dcd3772d7aee59fb0 (image=quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3, name=ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968-osd-2-deactivate, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=-18.2.1, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.29.1, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS, RELEASE=HEAD, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-05-04T12:44:35.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:35 smithi079 systemd[1]: ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968@osd.2.service: Deactivated successfully. 2024-05-04T12:44:35.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:35 smithi079 systemd[1]: Stopped Ceph osd.2 for d9b985c4-0a0e-11ef-bc97-c7b262605968. 2024-05-04T12:44:35.698 INFO:journalctl@ceph.osd.2.smithi079.stdout:May 04 12:44:35 smithi079 systemd[1]: ceph-d9b985c4-0a0e-11ef-bc97-c7b262605968@osd.2.service: Consumed 13min 30.521s CPU time. 2024-05-04T12:44:42.074 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-05-04T12:44:42.075 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-04T12:44:45.150 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.146 2024-05-04T12:44:54.152 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-05-04T12:44:54.153 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-04T12:44:57.214 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.146 2024-05-04T12:45:09.216 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-05-04T12:45:09.217 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-04T12:45:12.286 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.146 2024-05-04T12:45:15.287 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-05-04T12:45:15.288 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-04T12:45:15.358 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.146 2024-05-04T12:45:15.358 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-05-04T12:45:15.359 INFO:tasks.cephadm:Teardown complete 2024-05-04T12:45:15.359 ERROR:teuthology.run_tasks:Manager failed: cephadm Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_09dbd6bba52d19ba471c33bc15c008e6ad158ea6/qa/tasks/cephadm.py", line 1700, in distribute_config_and_admin_keyring yield File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_09dbd6bba52d19ba471c33bc15c008e6ad158ea6/qa/tasks/cephadm.py", line 2240, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_09dbd6bba52d19ba471c33bc15c008e6ad158ea6/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_ceph_ceph-c_09dbd6bba52d19ba471c33bc15c008e6ad158ea6/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_ceph_ceph-c_09dbd6bba52d19ba471c33bc15c008e6ad158ea6/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_09dbd6bba52d19ba471c33bc15c008e6ad158ea6/qa/tasks/cephadm.py", line 2243, in task log.info('Teardown begin') File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_09dbd6bba52d19ba471c33bc15c008e6ad158ea6/qa/tasks/cephadm.py", line 1833, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_09dbd6bba52d19ba471c33bc15c008e6ad158ea6/qa/tasks/cephadm.py", line 226, in download_cephadm _rm_cluster(ctx, cluster_name) File "/home/teuthworker/src/github.com_ceph_ceph-c_09dbd6bba52d19ba471c33bc15c008e6ad158ea6/qa/tasks/cephadm.py", line 395, in _rm_cluster remote.run(args=[ File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi146 2024-05-04T12:45:15.384 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-05-04T12:45:15.396 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_09dbd6bba52d19ba471c33bc15c008e6ad158ea6/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_ceph_ceph-c_09dbd6bba52d19ba471c33bc15c008e6ad158ea6/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_ceph_ceph-c_09dbd6bba52d19ba471c33bc15c008e6ad158ea6/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host 2024-05-04T12:45:15.412 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-05-04T12:45:15.412 DEBUG:teuthology.orchestra.run.smithi079:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-05-04T12:45:15.425 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-05-04T12:45:15.425 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-04T12:45:18.494 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.146 2024-05-04T12:45:27.495 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-05-04T12:45:27.496 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-04T12:45:30.558 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.146 2024-05-04T12:45:42.560 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-05-04T12:45:42.561 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-04T12:45:45.630 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.146 2024-05-04T12:45:48.632 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-05-04T12:45:48.633 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-04T12:45:48.706 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.146 2024-05-04T12:45:48.707 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-05-04T12:45:48.752 ERROR:teuthology.run_tasks:Manager failed: install Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/task/install/util.py", line 148, in ship_utilities yield File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_09dbd6bba52d19ba471c33bc15c008e6ad158ea6/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_ceph_ceph-c_09dbd6bba52d19ba471c33bc15c008e6ad158ea6/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_ceph_ceph-c_09dbd6bba52d19ba471c33bc15c008e6ad158ea6/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/task/install/__init__.py", line 626, in task yield File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/task/install/__init__.py", line 220, in install yield File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/task/install/util.py", line 150, in ship_utilities _remove_utilities(ctx, filenames) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/task/install/util.py", line 121, in _remove_utilities ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi146 2024-05-04T12:45:48.786 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-05-04T12:45:48.798 INFO:teuthology.task.clock:Checking final clock skew... 2024-05-04T12:45:48.799 DEBUG:teuthology.orchestra.run.smithi079:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-04T12:45:48.810 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-05-04T12:45:48.810 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-04T12:45:48.824 INFO:teuthology.orchestra.run.smithi079.stderr:bash: line 1: ntpq: command not found 2024-05-04T12:45:48.829 INFO:teuthology.orchestra.run.smithi079.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-04T12:45:48.829 INFO:teuthology.orchestra.run.smithi079.stdout:=============================================================================== 2024-05-04T12:45:48.829 INFO:teuthology.orchestra.run.smithi079.stdout:^* hv01.front.sepia.ceph.com 3 6 377 13 +1562us[+1562us] +/- 58ms 2024-05-04T12:45:48.829 INFO:teuthology.orchestra.run.smithi079.stdout:^+ hv02.front.sepia.ceph.com 3 7 377 16 -896us[ -896us] +/- 59ms 2024-05-04T12:45:48.829 INFO:teuthology.orchestra.run.smithi079.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 16 -1152us[-1152us] +/- 68ms 2024-05-04T12:45:48.830 INFO:teuthology.orchestra.run.smithi079.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-05-04T12:45:51.874 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.146 2024-05-04T12:46:00.876 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-05-04T12:46:00.877 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-04T12:46:03.934 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.146 2024-05-04T12:46:15.935 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-05-04T12:46:15.936 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-04T12:46:19.006 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.146 2024-05-04T12:46:22.007 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-05-04T12:46:22.008 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-04T12:46:22.082 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.146 2024-05-04T12:46:22.082 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-05-04T12:46:22.083 ERROR:teuthology.run_tasks:Manager failed: clock Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/task/clock.py", line 66, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_09dbd6bba52d19ba471c33bc15c008e6ad158ea6/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_ceph_ceph-c_09dbd6bba52d19ba471c33bc15c008e6ad158ea6/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_ceph_ceph-c_09dbd6bba52d19ba471c33bc15c008e6ad158ea6/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/task/clock.py", line 72, in task cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi146 2024-05-04T12:46:22.099 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-05-04T12:46:22.111 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-05-04T12:46:22.111 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-05-04T12:46:22.128 DEBUG:teuthology.orchestra.run.smithi079:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-05-04T12:46:22.208 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-05-04T12:46:22.209 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-04T12:46:25.278 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.146 2024-05-04T12:46:34.279 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-05-04T12:46:34.280 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-04T12:46:37.342 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.146 2024-05-04T12:46:49.343 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-05-04T12:46:49.344 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-04T12:46:52.414 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.146 2024-05-04T12:46:55.415 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-05-04T12:46:55.416 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-04T12:46:55.486 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.146 2024-05-04T12:46:55.487 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-05-04T12:46:55.487 ERROR:teuthology.run_tasks:Manager failed: selinux Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/task/__init__.py", line 136, in __exit__ self.teardown() File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/task/selinux.py", line 165, in teardown self.archive_log() File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/task/selinux.py", line 195, in archive_log self.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi146 2024-05-04T12:46:55.495 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-05-04T12:46:55.510 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-05-04T12:46:55.523 INFO:teuthology.task.internal:Duration was 2786.533675 seconds 2024-05-04T12:46:55.524 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-05-04T12:46:55.538 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-05-04T12:46:55.538 DEBUG:teuthology.orchestra.run.smithi079:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-05-04T12:46:55.544 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-05-04T12:46:55.544 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-04T12:46:55.585 INFO:teuthology.orchestra.run.smithi079.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-04T12:46:58.622 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.146 2024-05-04T12:47:07.624 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-05-04T12:47:07.625 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-04T12:47:10.690 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.146 2024-05-04T12:47:22.691 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-05-04T12:47:22.692 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-04T12:47:25.758 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.146 2024-05-04T12:47:28.760 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-05-04T12:47:28.760 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-04T12:47:28.834 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.146 2024-05-04T12:47:28.834 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-05-04T12:47:28.835 ERROR:teuthology.run_tasks:Manager failed: internal.syslog Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/task/internal/syslog.py", line 76, in syslog yield File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_09dbd6bba52d19ba471c33bc15c008e6ad158ea6/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_ceph_ceph-c_09dbd6bba52d19ba471c33bc15c008e6ad158ea6/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_ceph_ceph-c_09dbd6bba52d19ba471c33bc15c008e6ad158ea6/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/task/internal/syslog.py", line 85, in syslog cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi146 2024-05-04T12:47:28.851 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-05-04T12:47:28.866 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-05-04T12:47:28.867 DEBUG:teuthology.orchestra.run.smithi079:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-05-04T12:47:28.900 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-05-04T12:47:28.900 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-04T12:47:31.966 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.146 2024-05-04T12:47:40.967 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-05-04T12:47:40.968 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-04T12:47:44.030 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.146 2024-05-04T12:47:56.031 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-05-04T12:47:56.033 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-04T12:47:59.102 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.146 2024-05-04T12:48:02.104 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-05-04T12:48:02.105 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-04T12:48:02.174 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.146 2024-05-04T12:48:02.174 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-05-04T12:48:02.175 ERROR:teuthology.run_tasks:Manager failed: internal.sudo Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/task/internal/__init__.py", line 452, in sudo yield File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_09dbd6bba52d19ba471c33bc15c008e6ad158ea6/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_ceph_ceph-c_09dbd6bba52d19ba471c33bc15c008e6ad158ea6/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_ceph_ceph-c_09dbd6bba52d19ba471c33bc15c008e6ad158ea6/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/task/internal/__init__.py", line 455, in sudo ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi146 2024-05-04T12:48:02.191 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-05-04T12:48:02.204 DEBUG:teuthology.orchestra.run.smithi079:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-05-04T12:48:02.210 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-05-04T12:48:02.211 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-04T12:48:02.235 INFO:teuthology.orchestra.run.smithi079.stdout:kernel.core_pattern = core 2024-05-04T12:48:05.278 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.146 2024-05-04T12:48:14.279 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-05-04T12:48:14.280 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-04T12:48:17.342 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.146 2024-05-04T12:48:29.343 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-05-04T12:48:29.344 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-04T12:48:32.414 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.146 2024-05-04T12:48:35.415 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-05-04T12:48:35.416 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-04T12:48:35.486 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.146 2024-05-04T12:48:35.486 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-05-04T12:48:35.487 ERROR:teuthology.run_tasks:Manager failed: internal.coredump Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/task/internal/__init__.py", line 488, in coredump yield File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_09dbd6bba52d19ba471c33bc15c008e6ad158ea6/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_ceph_ceph-c_09dbd6bba52d19ba471c33bc15c008e6ad158ea6/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_ceph_ceph-c_09dbd6bba52d19ba471c33bc15c008e6ad158ea6/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/task/internal/__init__.py", line 492, in coredump cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi146 2024-05-04T12:48:35.504 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-05-04T12:48:35.519 INFO:teuthology.task.internal:Transferring archived files... 2024-05-04T12:48:35.520 DEBUG:teuthology.misc:Transferring archived files from smithi079:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-05-02_19:10:09-fs-wip-yuri4-testing-2024-04-29-0642-distro-default-smithi/7686275/remote/smithi079 2024-05-04T12:48:35.520 DEBUG:teuthology.orchestra.run.smithi079:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-05-04T12:48:35.560 DEBUG:teuthology.misc:Transferring archived files from smithi146:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-05-02_19:10:09-fs-wip-yuri4-testing-2024-04-29-0642-distro-default-smithi/7686275/remote/smithi146 2024-05-04T12:48:35.560 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-05-04T12:48:35.560 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-04T12:48:38.622 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.146 2024-05-04T12:48:47.623 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-05-04T12:48:47.624 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-04T12:48:50.689 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.146 2024-05-04T12:49:02.690 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-05-04T12:49:02.691 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-04T12:49:05.758 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.146 2024-05-04T12:49:08.760 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-05-04T12:49:08.761 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-04T12:49:08.834 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.146 2024-05-04T12:49:08.834 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-05-04T12:49:08.835 ERROR:teuthology.run_tasks:Manager failed: internal.archive Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/task/internal/__init__.py", line 392, in archive yield File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_09dbd6bba52d19ba471c33bc15c008e6ad158ea6/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_ceph_ceph-c_09dbd6bba52d19ba471c33bc15c008e6ad158ea6/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_ceph_ceph-c_09dbd6bba52d19ba471c33bc15c008e6ad158ea6/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/task/internal/__init__.py", line 418, in archive misc.pull_directory(rem, archive_dir, path, maybe_compress) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/misc.py", line 748, in pull_directory r = remote.get_tar_stream(remotedir, sudo=True, compress=False) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/remote.py", line 678, in get_tar_stream return self.run(args=args, wait=False, stdout=run.PIPE) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi146 2024-05-04T12:49:08.853 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-05-04T12:49:08.867 INFO:teuthology.task.internal:Not uploading archives. 2024-05-04T12:49:08.867 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-05-04T12:49:08.879 INFO:teuthology.task.internal:Tidying up after the test... 2024-05-04T12:49:08.879 DEBUG:teuthology.orchestra.run.smithi079:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-05-04T12:49:08.887 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-05-04T12:49:08.887 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-04T12:49:08.902 INFO:teuthology.orchestra.run.smithi079.stdout: 262213 4 drwxr-xr-x 3 ubuntu ubuntu 4096 May 4 12:45 /home/ubuntu/cephtest 2024-05-04T12:49:08.902 INFO:teuthology.orchestra.run.smithi079.stdout: 262304 212 -rwxr-xr-x 1 ubuntu ubuntu 215335 May 4 12:07 /home/ubuntu/cephtest/cephadm 2024-05-04T12:49:08.902 INFO:teuthology.orchestra.run.smithi079.stdout: 262214 4 drwxr-xr-x 4 ubuntu ubuntu 4096 May 4 12:48 /home/ubuntu/cephtest/archive 2024-05-04T12:49:08.902 INFO:teuthology.orchestra.run.smithi079.stdout: 524320 4 drwxr-xr-x 2 ubuntu ubuntu 4096 May 4 12:46 /home/ubuntu/cephtest/archive/audit 2024-05-04T12:49:08.903 INFO:teuthology.orchestra.run.smithi079.stdout: 524322 176 -rw------- 1 ubuntu ubuntu 179552 May 4 12:46 /home/ubuntu/cephtest/archive/audit/audit.log.gz 2024-05-04T12:49:08.903 INFO:teuthology.orchestra.run.smithi079.stdout: 262216 4 drwxr-xr-x 2 ubuntu ubuntu 4096 May 4 12:00 /home/ubuntu/cephtest/archive/syslog 2024-05-04T12:49:08.903 INFO:teuthology.orchestra.run.smithi079.stdout: 262218 0 -rw-rw-rw- 1 ubuntu ubuntu 0 May 4 12:00 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-04T12:49:08.903 INFO:teuthology.orchestra.run.smithi079.stdout: 262217 0 -rw-rw-rw- 1 ubuntu ubuntu 0 May 4 12:00 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-04T12:49:08.904 INFO:teuthology.orchestra.run.smithi079.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-05-04T12:49:11.966 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.146 2024-05-04T12:49:20.967 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-05-04T12:49:20.968 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-04T12:49:24.034 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.146 2024-05-04T12:49:36.035 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-05-04T12:49:36.036 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-04T12:49:39.102 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.146 2024-05-04T12:49:42.103 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-05-04T12:49:42.104 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-04T12:49:42.174 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.146 2024-05-04T12:49:42.174 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-05-04T12:49:42.175 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/task/internal/__init__.py", line 48, in base yield File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_09dbd6bba52d19ba471c33bc15c008e6ad158ea6/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_ceph_ceph-c_09dbd6bba52d19ba471c33bc15c008e6ad158ea6/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_ceph_ceph-c_09dbd6bba52d19ba471c33bc15c008e6ad158ea6/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/task/internal/__init__.py", line 54, in base ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi146 2024-05-04T12:49:42.192 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-05-04T12:49:42.204 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-05-04T12:49:42.231 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: OSError: [Errno 113] No route to host 2024-05-04T12:49:42.231 INFO:teuthology.nuke:Checking targets against current locks 2024-05-04T12:49:42.262 DEBUG:teuthology.nuke:shortname: smithi079 2024-05-04T12:49:42.263 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-05-04T12:49:42.286 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi079.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-05-02_19:10:09-fs-wip-yuri4-testing-2024-04-29-0642-distro-default-smithi/7686275', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-04 11:54:57.644276', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIHcuUETowTw9yhcydIBCjPslg9zshm19XIjmkFU3JrQdIY8ardoSS6N14RADKyAO/YwvweqfkZ6PWZ1qumUh1g='} 2024-05-04T12:49:42.290 DEBUG:teuthology.nuke:shortname: smithi146 2024-05-04T12:49:42.290 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-05-04T12:49:42.311 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi146.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-05-02_19:10:09-fs-wip-yuri4-testing-2024-04-29-0642-distro-default-smithi/7686275', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-04 11:54:57.645732', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIAm749bFz7aEejwdZa56XYzAwBSFHJAm+KZD1iGYlzY815cjRcRs8hktFJusOtxnspIppe+lxO7fvJzjCBOFzs='} 2024-05-04T12:49:42.339 INFO:teuthology.orchestra.console.smithi079:Power off 2024-05-04T12:49:42.340 DEBUG:teuthology.orchestra.console.smithi079:pexpect command: ipmitool -H smithi079.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-05-04T12:49:42.363 DEBUG:teuthology.orchestra.console.smithi079:power off output: Chassis Power Control: Down/Off 2024-05-04T12:49:42.364 DEBUG:teuthology.orchestra.console.smithi079:pexpect command: ipmitool -H smithi079.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-04T12:49:42.381 INFO:teuthology.orchestra.console.smithi146:Power off 2024-05-04T12:49:42.381 DEBUG:teuthology.orchestra.console.smithi146:pexpect command: ipmitool -H smithi146.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-05-04T12:49:42.391 DEBUG:teuthology.orchestra.console.smithi079:check power output: Chassis Power is on 2024-05-04T12:49:42.404 DEBUG:teuthology.orchestra.console.smithi146:power off output: Chassis Power Control: Down/Off 2024-05-04T12:49:42.404 DEBUG:teuthology.orchestra.console.smithi146:pexpect command: ipmitool -H smithi146.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-04T12:49:42.428 DEBUG:teuthology.orchestra.console.smithi146:check power output: Chassis Power is on 2024-05-04T12:49:46.393 DEBUG:teuthology.orchestra.console.smithi079:pexpect command: ipmitool -H smithi079.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-04T12:49:46.429 DEBUG:teuthology.orchestra.console.smithi146:pexpect command: ipmitool -H smithi146.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-04T12:49:46.504 DEBUG:teuthology.orchestra.console.smithi079:check power output: Chassis Power is on 2024-05-04T12:49:46.540 DEBUG:teuthology.orchestra.console.smithi146:check power output: Chassis Power is on 2024-05-04T12:49:50.509 DEBUG:teuthology.orchestra.console.smithi079:pexpect command: ipmitool -H smithi079.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-04T12:49:50.541 DEBUG:teuthology.orchestra.console.smithi146:pexpect command: ipmitool -H smithi146.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-04T12:49:50.619 DEBUG:teuthology.orchestra.console.smithi079:check power output: Chassis Power is off 2024-05-04T12:49:50.652 DEBUG:teuthology.orchestra.console.smithi146:check power output: Chassis Power is on 2024-05-04T12:49:50.721 INFO:teuthology.orchestra.console.smithi079:Power off completed 2024-05-04T12:49:54.655 DEBUG:teuthology.orchestra.console.smithi146:pexpect command: ipmitool -H smithi146.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-04T12:49:54.766 DEBUG:teuthology.orchestra.console.smithi146:check power output: Chassis Power is off 2024-05-04T12:49:54.867 INFO:teuthology.orchestra.console.smithi146:Power off completed 2024-05-04T12:49:54.972 INFO:teuthology.run:Summary data: description: fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.1} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} duration: 2786.5336747169495 failure_reason: 'Command failed on smithi079 with status 1: "sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d9b985c4-0a0e-11ef-bc97-c7b262605968 -e sha1=09dbd6bba52d19ba471c33bc15c008e6ad158ea6 -- bash -c ''ceph orch ps''"' flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=3eaf4662403647c6bbbc16e6b83a74fb status: fail success: false 2024-05-04T12:49:54.973 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-05-04T12:49:55.050 INFO:teuthology.run:FAIL